[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:05:41 dist-test-slave-2p7j krb5kdc[3947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:05:41 dist-test-slave-2p7j krb5kdc[3947](info): set up 2 sockets
Sep 24 07:05:41 dist-test-slave-2p7j krb5kdc[3947](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 07:05:42.047925 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:05:42 dist-test-slave-2p7j krb5kdc[3947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:05:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:05:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:05:42.178712 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:46153
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:41801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:46153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:05:42.291215 3960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:05:42.291842 3960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:05:42.291945 3960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:05:42.297360 3960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:05:42.297418 3960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:05:42.297438 3960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:05:42.297454 3960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:05:42.297468 3960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:05:42.302847 3960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:41801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:46153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:46153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:05:42.303860 3960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:05:42.304802 3960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:05:42.313373 3960 server_base.cc:1034] running on GCE node
W20260924 07:05:42.313421 3968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:42.313488 3966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:42.313560 3965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:05:42.314260 3960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:05:42.315292 3960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:05:42.316462 3960 hybrid_clock.cc:648] HybridClock initialized: now 1790233542316443 us; error 50 us; skew 500 ppm
Sep 24 07:05:42 dist-test-slave-2p7j krb5kdc[3947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233542, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:05:42.319193 3960 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:05:42.320062 3960 webserver.cc:460] Webserver started at http://127.3.217.62:41389/ using document root <none> and password file <none>
I20260924 07:05:42.320570 3960 fs_manager.cc:359] Metadata directory not provided
I20260924 07:05:42.320633 3960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:05:42.320817 3960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:05:42.322728 3960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "587e3fefbb9444c08ef9c14645dfff05"
format_stamp: "Formatted at 2026-09-24 07:05:42 on dist-test-slave-2p7j"
server_key: "e3025230019ed68f4aeccfa05d81e791"
server_key_iv: "02d23d79f8952e483c08a413c906efbb"
server_key_version: "encryptionkey@0"
I20260924 07:05:42.323248 3960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "587e3fefbb9444c08ef9c14645dfff05"
format_stamp: "Formatted at 2026-09-24 07:05:42 on dist-test-slave-2p7j"
server_key: "e3025230019ed68f4aeccfa05d81e791"
server_key_iv: "02d23d79f8952e483c08a413c906efbb"
server_key_version: "encryptionkey@0"
I20260924 07:05:42.326747 3960 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:05:42.329665 3975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:42.331452 3960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:05:42.331712 3960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "587e3fefbb9444c08ef9c14645dfff05"
format_stamp: "Formatted at 2026-09-24 07:05:42 on dist-test-slave-2p7j"
server_key: "e3025230019ed68f4aeccfa05d81e791"
server_key_iv: "02d23d79f8952e483c08a413c906efbb"
server_key_version: "encryptionkey@0"
I20260924 07:05:42.331857 3960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:05:42.348585 3960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:05:42.352279 3960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:05:42.352458 3960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:05:42.359418 3960 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:46153
I20260924 07:05:42.359469 4027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:46153 every 8 connection(s)
I20260924 07:05:42.360327 3960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:05:42.362977 4028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:42.366706 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 3960
I20260924 07:05:42.366849 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:05:42.368105 3940 external_mini_cluster.cc:1428] Setting key c928781a2bb4fca560c6e58a77abcdbb
I20260924 07:05:42.368479 4028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 587e3fefbb9444c08ef9c14645dfff05: Bootstrap starting.
I20260924 07:05:42.370929 4028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 587e3fefbb9444c08ef9c14645dfff05: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:42.371754 4028 log.cc:824] T 00000000000000000000000000000000 P 587e3fefbb9444c08ef9c14645dfff05: Log is configured to *not* fsync() on all Append() calls
I20260924 07:05:42.373821 4028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 587e3fefbb9444c08ef9c14645dfff05: No bootstrap required, opened a new log
I20260924 07:05:42.377909 4028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 587e3fefbb9444c08ef9c14645dfff05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587e3fefbb9444c08ef9c14645dfff05" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 46153 } }
I20260924 07:05:42.378119 4028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 587e3fefbb9444c08ef9c14645dfff05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:42.378168 4028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 587e3fefbb9444c08ef9c14645dfff05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 587e3fefbb9444c08ef9c14645dfff05, State: Initialized, Role: FOLLOWER
I20260924 07:05:42.378652 4028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 587e3fefbb9444c08ef9c14645dfff05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587e3fefbb9444c08ef9c14645dfff05" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 46153 } }
I20260924 07:05:42.378821 4028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 587e3fefbb9444c08ef9c14645dfff05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:05:42.378876 4028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 587e3fefbb9444c08ef9c14645dfff05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:05:42.378970 4028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 587e3fefbb9444c08ef9c14645dfff05 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:42.380048 4028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 587e3fefbb9444c08ef9c14645dfff05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587e3fefbb9444c08ef9c14645dfff05" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 46153 } }
I20260924 07:05:42.380394 4028 leader_election.cc:302] T 00000000000000000000000000000000 P 587e3fefbb9444c08ef9c14645dfff05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 587e3fefbb9444c08ef9c14645dfff05; no voters:
I20260924 07:05:42.380740 4028 leader_election.cc:288] T 00000000000000000000000000000000 P 587e3fefbb9444c08ef9c14645dfff05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:05:42.380820 4033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 587e3fefbb9444c08ef9c14645dfff05 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:05:42.381031 4033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 587e3fefbb9444c08ef9c14645dfff05 [term 1 LEADER]: Becoming Leader. State: Replica: 587e3fefbb9444c08ef9c14645dfff05, State: Running, Role: LEADER
I20260924 07:05:42.381682 4033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 587e3fefbb9444c08ef9c14645dfff05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587e3fefbb9444c08ef9c14645dfff05" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 46153 } }
I20260924 07:05:42.382056 4028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 587e3fefbb9444c08ef9c14645dfff05 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:05:42.383131 4034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 587e3fefbb9444c08ef9c14645dfff05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "587e3fefbb9444c08ef9c14645dfff05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587e3fefbb9444c08ef9c14645dfff05" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 46153 } } }
I20260924 07:05:42.383252 4034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 587e3fefbb9444c08ef9c14645dfff05 [sys.catalog]: This master's current role is: LEADER
I20260924 07:05:42.383598 4035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 587e3fefbb9444c08ef9c14645dfff05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 587e3fefbb9444c08ef9c14645dfff05. Latest consensus state: current_term: 1 leader_uuid: "587e3fefbb9444c08ef9c14645dfff05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587e3fefbb9444c08ef9c14645dfff05" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 46153 } } }
I20260924 07:05:42.383683 4035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 587e3fefbb9444c08ef9c14645dfff05 [sys.catalog]: This master's current role is: LEADER
I20260924 07:05:42.384037 4042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:05:42.387434 4042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 07:05:42 dist-test-slave-2p7j krb5kdc[3947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:05:42.393175 4042 catalog_manager.cc:1347] Generated new cluster ID: c23440ff39a14e93ac8866a2db46bca2
I20260924 07:05:42.393318 4042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:05:42.400000 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:42.370787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35308 (local address 127.3.217.62:46153)
0924 07:05:42.371272 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:42.371290 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:42.372383 (+ 1093us) server_negotiation.cc:407] Connection header received
0924 07:05:42.377577 (+ 5194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:42.377597 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:42.377904 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:42.378236 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:42.381062 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:42.382562 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:42.384501 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:42.385170 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:42.385289 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:42.391170 (+ 5881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:42.391181 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:42.391208 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:42.391263 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:42.394727 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:42.395336 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:42.395342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:42.395348 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:42.395554 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:42.395943 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:42.395946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:42.395948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:42.396405 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:42.396577 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:42.397481 (+ 904us) server_negotiation.cc:299] Negotiation successful
0924 07:05:42.399111 (+ 1630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":113,"threads_started":1}
I20260924 07:05:42.402827 4042 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:05:42.403928 4042 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:05:42.418745 4042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 587e3fefbb9444c08ef9c14645dfff05: Generated new TSK 0
I20260924 07:05:42.419389 4042 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
I20260924 07:05:42.451143 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:46153
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:41801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:05:42.558146 4054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:05:42.558370 4054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:05:42.558410 4054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:05:42.562331 4054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:05:42.562381 4054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:05:42.562438 4054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:05:42.567449 4054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:41801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:46153
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:05:42.568341 4054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:05:42.569242 4054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:05:42.578186 4060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:42.578253 4059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:42.578191 4062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:05:42.578368 4054 server_base.cc:1034] running on GCE node
I20260924 07:05:42.578805 4054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:05:42.579361 4054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:05:42.580536 4054 hybrid_clock.cc:648] HybridClock initialized: now 1790233542580506 us; error 45 us; skew 500 ppm
Sep 24 07:05:42 dist-test-slave-2p7j krb5kdc[3947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233542, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:05:42.583380 4054 init.cc:377] Logged in from keytab as kudu/127.3.217.1@KRBTEST.COM (short username kudu)
I20260924 07:05:42.584214 4054 webserver.cc:460] Webserver started at http://127.3.217.1:40501/ using document root <none> and password file <none>
I20260924 07:05:42.584705 4054 fs_manager.cc:359] Metadata directory not provided
I20260924 07:05:42.584760 4054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:05:42.584920 4054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:05:42.586813 4054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "f05f8b8e4d0e45c28f8e77bd203471e9"
format_stamp: "Formatted at 2026-09-24 07:05:42 on dist-test-slave-2p7j"
server_key: "7b28f4526be35b3466ba8273bbf95df5"
server_key_iv: "db0a02ad1ae0e451d927a8122b4d8bd6"
server_key_version: "encryptionkey@0"
I20260924 07:05:42.587292 4054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "f05f8b8e4d0e45c28f8e77bd203471e9"
format_stamp: "Formatted at 2026-09-24 07:05:42 on dist-test-slave-2p7j"
server_key: "7b28f4526be35b3466ba8273bbf95df5"
server_key_iv: "db0a02ad1ae0e451d927a8122b4d8bd6"
server_key_version: "encryptionkey@0"
I20260924 07:05:42.591084 4054 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 07:05:42.593423 4069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:42.594595 4054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:05:42.594744 4054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "f05f8b8e4d0e45c28f8e77bd203471e9"
format_stamp: "Formatted at 2026-09-24 07:05:42 on dist-test-slave-2p7j"
server_key: "7b28f4526be35b3466ba8273bbf95df5"
server_key_iv: "db0a02ad1ae0e451d927a8122b4d8bd6"
server_key_version: "encryptionkey@0"
I20260924 07:05:42.594835 4054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:05:42.605863 4054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:05:42.608609 4054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:05:42.608767 4054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:05:42.609257 4054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:05:42.609982 4054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:05:42.610028 4054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:42.610076 4054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:05:42.610105 4054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:42.618515 4054 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:44929
I20260924 07:05:42.618637 4182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:44929 every 8 connection(s)
I20260924 07:05:42.619294 4054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:05:42.626262 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 4054
I20260924 07:05:42.626356 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:05:42.626575 3940 external_mini_cluster.cc:1428] Setting key 5102de7841c9711e4c90a85991d377df
Sep 24 07:05:42 dist-test-slave-2p7j krb5kdc[3947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233542, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:05:42.635188 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:42.621138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:50039 (local address 127.3.217.62:46153)
0924 07:05:42.621271 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:42.621276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:42.622058 (+ 782us) server_negotiation.cc:407] Connection header received
0924 07:05:42.623592 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:42.623596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:42.623682 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:42.623775 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:42.625038 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:42.625583 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:42.626764 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:42.627081 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:42.627171 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:42.630952 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:42.630956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:42.630959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:42.631370 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:42.632727 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:42.633393 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:42.633400 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:42.633401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:42.633558 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:42.634024 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:42.634028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:42.634029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:42.634236 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:42.634318 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:42.634901 (+ 583us) server_negotiation.cc:299] Negotiation successful
0924 07:05:42.635032 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:05:42.636278 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:42.621487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:46153 (local address 127.3.217.1:50039)
0924 07:05:42.621921 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:42.621959 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:42.623175 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:42.623957 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:42.623965 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:42.624409 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:05:42.624886 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:42.624899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:42.625707 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:42.625710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:42.626651 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:42.626662 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:42.627243 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:42.627262 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:42.627450 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:42.628248 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:05:42.628268 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:05:42.630717 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:05:42.632840 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:42.632846 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:42.632859 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:42.633206 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:42.633712 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:42.633716 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:42.633718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:42.633895 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:42.634339 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:05:42.634345 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:05:42.634630 (+ 285us) client_negotiation.cc:769] Sending connection context
0924 07:05:42.634835 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 07:05:42.635168 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for kudu/127.3.217.2@KRBTEST.COM; defaulting to no policy
I20260924 07:05:42.637650 4183 heartbeater.cc:347] Connected to a master server at 127.3.217.62:46153
I20260924 07:05:42.637871 4183 heartbeater.cc:464] Registering TS with master...
I20260924 07:05:42.638289 4183 heartbeater.cc:511] Master 127.3.217.62:46153 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.2@KRBTEST.COM" created.
I20260924 07:05:42.639932 3992 ts_manager.cc:194] Registered new tserver with Master: f05f8b8e4d0e45c28f8e77bd203471e9 (127.3.217.1:44929)
I20260924 07:05:42.641660 3992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.1@KRBTEST.COM'} at 127.3.217.1:50039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
I20260924 07:05:42.655167 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:46153
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:41801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:05:42.766498 4188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:05:42.766732 4188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:05:42.766779 4188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:05:42.770901 4188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:05:42.770947 4188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:05:42.771013 4188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:05:42.776275 4188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:41801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:46153
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:05:42.777225 4188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:05:42.778640 4188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:05:42.789144 4196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:42.789201 4194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:05:42.789422 4188 server_base.cc:1034] running on GCE node
W20260924 07:05:42.789175 4193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:05:42.789954 4188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:05:42.790680 4188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:05:42.791855 4188 hybrid_clock.cc:648] HybridClock initialized: now 1790233542791832 us; error 32 us; skew 500 ppm
Sep 24 07:05:42 dist-test-slave-2p7j krb5kdc[3947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233542, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:05:42.795137 4188 init.cc:377] Logged in from keytab as kudu/127.3.217.2@KRBTEST.COM (short username kudu)
I20260924 07:05:42.796185 4188 webserver.cc:460] Webserver started at http://127.3.217.2:37977/ using document root <none> and password file <none>
I20260924 07:05:42.796846 4188 fs_manager.cc:359] Metadata directory not provided
I20260924 07:05:42.796928 4188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:05:42.797135 4188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:05:42.800086 4188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "754cb8664b564472b795e67219e06b88"
format_stamp: "Formatted at 2026-09-24 07:05:42 on dist-test-slave-2p7j"
server_key: "632463050141e862178957d8c7fdcea9"
server_key_iv: "6542a28bb39e9165517ced11d9c88a95"
server_key_version: "encryptionkey@0"
I20260924 07:05:42.800839 4188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "754cb8664b564472b795e67219e06b88"
format_stamp: "Formatted at 2026-09-24 07:05:42 on dist-test-slave-2p7j"
server_key: "632463050141e862178957d8c7fdcea9"
server_key_iv: "6542a28bb39e9165517ced11d9c88a95"
server_key_version: "encryptionkey@0"
I20260924 07:05:42.805351 4188 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:05:42.807586 4203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:42.808753 4188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:05:42.808843 4188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "754cb8664b564472b795e67219e06b88"
format_stamp: "Formatted at 2026-09-24 07:05:42 on dist-test-slave-2p7j"
server_key: "632463050141e862178957d8c7fdcea9"
server_key_iv: "6542a28bb39e9165517ced11d9c88a95"
server_key_version: "encryptionkey@0"
I20260924 07:05:42.808943 4188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:05:42.831061 4188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:05:42.834048 4188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:05:42.834239 4188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:05:42.834811 4188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:05:42.836138 4188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:05:42.836200 4188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:42.836256 4188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:05:42.836287 4188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:42.845093 4188 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:36249
I20260924 07:05:42.845181 4316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:36249 every 8 connection(s)
I20260924 07:05:42.845966 4188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:05:42.851774 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 4188
I20260924 07:05:42.851894 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:05:42.852181 3940 external_mini_cluster.cc:1428] Setting key 490e492f2b6bc2483da37df2edd7e483
Sep 24 07:05:42 dist-test-slave-2p7j krb5kdc[3947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233542, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:05:42.860492 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:42.847773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:43973 (local address 127.3.217.62:46153)
0924 07:05:42.847911 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:42.847915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:42.848727 (+ 812us) server_negotiation.cc:407] Connection header received
0924 07:05:42.850082 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:42.850085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:42.850129 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:42.850202 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:42.851841 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:42.852443 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:42.854051 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:42.854278 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:42.854365 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:42.856977 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:42.856980 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:42.856982 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:42.857002 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:42.858321 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:42.858778 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:42.858781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:42.858783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:42.858869 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:42.859290 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:42.859294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:42.859296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:42.859570 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:42.859666 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:42.860263 (+ 597us) server_negotiation.cc:299] Negotiation successful
0924 07:05:42.860371 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.3.217.3@KRBTEST.COM; defaulting to no policy
I20260924 07:05:42.861573 4319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:42.848124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:46153 (local address 127.3.217.2:43973)
0924 07:05:42.848559 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:42.848625 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:42.849866 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:42.850358 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:42.850365 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:42.850733 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:05:42.851685 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:42.851709 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:42.852565 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:42.852569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:42.853916 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:42.853926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:42.854642 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:42.854646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:42.854838 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:42.855473 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:05:42.855488 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:05:42.856832 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:05:42.858435 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:42.858441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:42.858454 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:42.858688 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:42.858986 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:42.858990 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:42.858992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:42.859189 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:42.859681 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:05:42.859688 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:05:42.859996 (+ 308us) client_negotiation.cc:769] Sending connection context
0924 07:05:42.860217 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 07:05:42.860538 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":95,"threads_started":1}
I20260924 07:05:42.862632 4317 heartbeater.cc:347] Connected to a master server at 127.3.217.62:46153
I20260924 07:05:42.862855 4317 heartbeater.cc:464] Registering TS with master...
I20260924 07:05:42.863373 4317 heartbeater.cc:511] Master 127.3.217.62:46153 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.3@KRBTEST.COM" created.
I20260924 07:05:42.865388 3992 ts_manager.cc:194] Registered new tserver with Master: 754cb8664b564472b795e67219e06b88 (127.3.217.2:36249)
I20260924 07:05:42.866006 3992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.2@KRBTEST.COM'} at 127.3.217.2:43973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
I20260924 07:05:42.879621 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:46153
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:41801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:05:43.006502 4322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:05:43.006742 4322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:05:43.006820 4322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:05:43.011123 4322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:05:43.011191 4322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:05:43.011252 4322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:05:43.017236 4322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:41801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:46153
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:05:43.018193 4322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:05:43.019125 4322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:05:43.028465 4328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:43.028446 4327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:43.028460 4330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:05:43.028769 4322 server_base.cc:1034] running on GCE node
I20260924 07:05:43.029165 4322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:05:43.029789 4322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:05:43.030961 4322 hybrid_clock.cc:648] HybridClock initialized: now 1790233543030943 us; error 26 us; skew 500 ppm
Sep 24 07:05:43 dist-test-slave-2p7j krb5kdc[3947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233543, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:05:43.033597 4322 init.cc:377] Logged in from keytab as kudu/127.3.217.3@KRBTEST.COM (short username kudu)
I20260924 07:05:43.034412 4322 webserver.cc:460] Webserver started at http://127.3.217.3:44205/ using document root <none> and password file <none>
I20260924 07:05:43.035107 4322 fs_manager.cc:359] Metadata directory not provided
I20260924 07:05:43.035180 4322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:05:43.035341 4322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:05:43.037297 4322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "35c45c4d895a49549bb2040f3b73515c"
format_stamp: "Formatted at 2026-09-24 07:05:43 on dist-test-slave-2p7j"
server_key: "db37c05ae91026de18f0833e77f128f0"
server_key_iv: "5efdc0e743e8100fab59f7e6f9147e42"
server_key_version: "encryptionkey@0"
I20260924 07:05:43.037770 4322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "35c45c4d895a49549bb2040f3b73515c"
format_stamp: "Formatted at 2026-09-24 07:05:43 on dist-test-slave-2p7j"
server_key: "db37c05ae91026de18f0833e77f128f0"
server_key_iv: "5efdc0e743e8100fab59f7e6f9147e42"
server_key_version: "encryptionkey@0"
I20260924 07:05:43.041219 4322 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:05:43.043751 4337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:43.045361 4322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:05:43.045498 4322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "35c45c4d895a49549bb2040f3b73515c"
format_stamp: "Formatted at 2026-09-24 07:05:43 on dist-test-slave-2p7j"
server_key: "db37c05ae91026de18f0833e77f128f0"
server_key_iv: "5efdc0e743e8100fab59f7e6f9147e42"
server_key_version: "encryptionkey@0"
I20260924 07:05:43.045585 4322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:05:43.069185 4322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:05:43.072316 4322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:05:43.072486 4322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:05:43.073040 4322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:05:43.073813 4322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:05:43.073858 4322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:43.073909 4322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:05:43.073938 4322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:43.084946 4322 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:35907
I20260924 07:05:43.085035 4450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:35907 every 8 connection(s)
I20260924 07:05:43.085815 4322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
Sep 24 07:05:43 dist-test-slave-2p7j krb5kdc[3947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233543, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:05:43.095528 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 4322
I20260924 07:05:43.095674 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:05:43.095911 3940 external_mini_cluster.cc:1428] Setting key f11dea70c33a0cf432daa9145ddb02da
I20260924 07:05:43.099984 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:43.087569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:45517 (local address 127.3.217.62:46153)
0924 07:05:43.087701 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:43.087705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:43.088480 (+ 775us) server_negotiation.cc:407] Connection header received
0924 07:05:43.089773 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:43.089777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:43.089829 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:43.089901 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:43.090975 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:43.091530 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:43.092618 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:43.092839 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:43.092922 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:43.095845 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:43.095851 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:43.095854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:43.095888 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:43.097560 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:43.098311 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:43.098316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:43.098318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:43.098497 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:43.098900 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:43.098904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:43.098905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:43.099155 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:43.099239 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:43.099753 (+ 514us) server_negotiation.cc:299] Negotiation successful
0924 07:05:43.099862 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:05:43.100688 4453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:43.087912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:46153 (local address 127.3.217.3:45517)
0924 07:05:43.088340 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:43.088379 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:43.089548 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:43.090056 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:43.090063 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:43.090391 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:05:43.090845 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:43.090857 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:43.091638 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:43.091641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:43.092505 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:43.092514 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:43.092964 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:43.092968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:43.093124 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:43.093741 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:05:43.093783 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:05:43.095485 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:05:43.097678 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:43.097686 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:43.097708 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:43.098178 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:43.098624 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:43.098628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:43.098630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:43.098788 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:43.099252 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:05:43.099257 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:05:43.099477 (+ 220us) client_negotiation.cc:769] Sending connection context
0924 07:05:43.099683 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 07:05:43.099931 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":96,"threads_started":1}
I20260924 07:05:43.102007 4451 heartbeater.cc:347] Connected to a master server at 127.3.217.62:46153
I20260924 07:05:43.102376 4451 heartbeater.cc:464] Registering TS with master...
I20260924 07:05:43.103086 4451 heartbeater.cc:511] Master 127.3.217.62:46153 requested a full tablet report, sending...
I20260924 07:05:43.104526 3992 ts_manager.cc:194] Registered new tserver with Master: 35c45c4d895a49549bb2040f3b73515c (127.3.217.3:35907)
I20260924 07:05:43.105232 3992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.3@KRBTEST.COM'} at 127.3.217.3:45517
I20260924 07:05:43.111488 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 07:05:43 dist-test-slave-2p7j krb5kdc[3947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233543, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 07:05:43 dist-test-slave-2p7j krb5kdc[3947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233543, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.217.1@KRBTEST.COM
I20260924 07:05:43.139621 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:43.129156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.3.217.1:44929)
0924 07:05:43.129408 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:43.129422 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:43.129444 (+ 22us) server_negotiation.cc:407] Connection header received
0924 07:05:43.129492 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:43.129496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:43.129671 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:43.129787 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:43.130468 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:43.131253 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:43.132179 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:43.132565 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:43.132652 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:43.135781 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:43.135790 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:43.135802 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:43.135827 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:43.137578 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:43.138192 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:43.138198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:43.138199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:43.138290 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:43.138667 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:43.138671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:43.138672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:43.138914 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:43.139085 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:43.139307 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 07:05:43.139441 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":72,"threads_started":1}
W20260924 07:05:43.140666 4096 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49932
Sep 24 07:05:43 dist-test-slave-2p7j krb5kdc[3947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 07:05:43 dist-test-slave-2p7j krb5kdc[3947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.1@KRBTEST.COM
I20260924 07:05:43.167934 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:43.158660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49948 (local address 127.3.217.1:44929)
0924 07:05:43.159053 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:43.159068 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:43.159121 (+ 53us) server_negotiation.cc:407] Connection header received
0924 07:05:43.159181 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:43.159183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:43.159243 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:43.159316 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:43.160080 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:43.160615 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:43.161498 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:43.161783 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:43.161905 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:43.164233 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:43.164236 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:43.164238 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:43.164273 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:43.166018 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:43.166510 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:43.166514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:43.166516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:43.166624 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:43.166993 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:43.166996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:43.166998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:43.167210 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:43.167333 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:43.167634 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 07:05:43.167769 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 07:05:43.169103 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 4054
I20260924 07:05:43.174852 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 4188
I20260924 07:05:43.181614 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 4322
I20260924 07:05:43.186612 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 3960
2026-09-24T07:05:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2126 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:05:43 dist-test-slave-2p7j krb5kdc[4465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:05:43 dist-test-slave-2p7j krb5kdc[4465](info): set up 2 sockets
Sep 24 07:05:43 dist-test-slave-2p7j krb5kdc[4465](info): commencing operation
krb5kdc: starting...
W20260924 07:05:44.156111 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:05:44 dist-test-slave-2p7j krb5kdc[4465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:05:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:05:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:05:44.275977 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:33507
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:33507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:05:44.377844 4478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:05:44.378055 4478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:05:44.378091 4478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:05:44.381847 4478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:05:44.381889 4478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:05:44.381906 4478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:05:44.381920 4478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:05:44.381933 4478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:05:44.386776 4478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:33507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:33507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:05:44.387594 4478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:05:44.388424 4478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:05:44.395918 4486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:44.395925 4483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:44.395925 4484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:05:44.396219 4478 server_base.cc:1034] running on GCE node
I20260924 07:05:44.396682 4478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:05:44.397624 4478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:05:44.398840 4478 hybrid_clock.cc:648] HybridClock initialized: now 1790233544398770 us; error 37 us; skew 500 ppm
Sep 24 07:05:44 dist-test-slave-2p7j krb5kdc[4465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233544, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:05:44.402395 4478 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:05:44.403245 4478 webserver.cc:460] Webserver started at http://127.3.217.62:35851/ using document root <none> and password file <none>
I20260924 07:05:44.403795 4478 fs_manager.cc:359] Metadata directory not provided
I20260924 07:05:44.403858 4478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:05:44.404040 4478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:05:44.405879 4478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "dae868f47c73474fb9e338d3e7bd6f5a"
format_stamp: "Formatted at 2026-09-24 07:05:44 on dist-test-slave-2p7j"
server_key: "1b9fa8a0616ed8c28aecbb10032e38e6"
server_key_iv: "e8a627795ab49501c17ce60548f3e2b2"
server_key_version: "encryptionkey@0"
I20260924 07:05:44.406368 4478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "dae868f47c73474fb9e338d3e7bd6f5a"
format_stamp: "Formatted at 2026-09-24 07:05:44 on dist-test-slave-2p7j"
server_key: "1b9fa8a0616ed8c28aecbb10032e38e6"
server_key_iv: "e8a627795ab49501c17ce60548f3e2b2"
server_key_version: "encryptionkey@0"
I20260924 07:05:44.409833 4478 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:05:44.412055 4493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:44.413406 4478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:05:44.413520 4478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "dae868f47c73474fb9e338d3e7bd6f5a"
format_stamp: "Formatted at 2026-09-24 07:05:44 on dist-test-slave-2p7j"
server_key: "1b9fa8a0616ed8c28aecbb10032e38e6"
server_key_iv: "e8a627795ab49501c17ce60548f3e2b2"
server_key_version: "encryptionkey@0"
I20260924 07:05:44.413616 4478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:05:44.424726 4478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:05:44.427393 4478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:05:44.427583 4478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:05:44.434305 4478 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:33507
I20260924 07:05:44.434331 4545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:33507 every 8 connection(s)
I20260924 07:05:44.435148 4478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:05:44.437664 4546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:44.441159 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 4478
I20260924 07:05:44.441256 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:05:44.441464 3940 external_mini_cluster.cc:1428] Setting key 31b5828a4b44f2e8a0c6913a290412cc
I20260924 07:05:44.443289 4546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dae868f47c73474fb9e338d3e7bd6f5a: Bootstrap starting.
I20260924 07:05:44.446017 4546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dae868f47c73474fb9e338d3e7bd6f5a: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:44.446838 4546 log.cc:824] T 00000000000000000000000000000000 P dae868f47c73474fb9e338d3e7bd6f5a: Log is configured to *not* fsync() on all Append() calls
I20260924 07:05:44.448783 4546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dae868f47c73474fb9e338d3e7bd6f5a: No bootstrap required, opened a new log
Sep 24 07:05:44 dist-test-slave-2p7j krb5kdc[4465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:05:44.452459 4546 raft_consensus.cc:348] T 00000000000000000000000000000000 P dae868f47c73474fb9e338d3e7bd6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae868f47c73474fb9e338d3e7bd6f5a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33507 } }
I20260924 07:05:44.452725 4546 raft_consensus.cc:374] T 00000000000000000000000000000000 P dae868f47c73474fb9e338d3e7bd6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:44.452781 4546 raft_consensus.cc:729] T 00000000000000000000000000000000 P dae868f47c73474fb9e338d3e7bd6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dae868f47c73474fb9e338d3e7bd6f5a, State: Initialized, Role: FOLLOWER
I20260924 07:05:44.453203 4546 consensus_queue.cc:260] T 00000000000000000000000000000000 P dae868f47c73474fb9e338d3e7bd6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae868f47c73474fb9e338d3e7bd6f5a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33507 } }
I20260924 07:05:44.453339 4546 raft_consensus.cc:388] T 00000000000000000000000000000000 P dae868f47c73474fb9e338d3e7bd6f5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:05:44.453406 4546 raft_consensus.cc:482] T 00000000000000000000000000000000 P dae868f47c73474fb9e338d3e7bd6f5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:05:44.453506 4546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dae868f47c73474fb9e338d3e7bd6f5a [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:44.454455 4546 raft_consensus.cc:504] T 00000000000000000000000000000000 P dae868f47c73474fb9e338d3e7bd6f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae868f47c73474fb9e338d3e7bd6f5a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33507 } }
I20260924 07:05:44.454792 4546 leader_election.cc:302] T 00000000000000000000000000000000 P dae868f47c73474fb9e338d3e7bd6f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dae868f47c73474fb9e338d3e7bd6f5a; no voters:
I20260924 07:05:44.455117 4546 leader_election.cc:288] T 00000000000000000000000000000000 P dae868f47c73474fb9e338d3e7bd6f5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:05:44.455196 4551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dae868f47c73474fb9e338d3e7bd6f5a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:05:44.455395 4551 raft_consensus.cc:686] T 00000000000000000000000000000000 P dae868f47c73474fb9e338d3e7bd6f5a [term 1 LEADER]: Becoming Leader. State: Replica: dae868f47c73474fb9e338d3e7bd6f5a, State: Running, Role: LEADER
I20260924 07:05:44.456138 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:44.442862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53744 (local address 127.3.217.62:33507)
0924 07:05:44.443247 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:44.443264 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:44.443304 (+ 40us) server_negotiation.cc:407] Connection header received
0924 07:05:44.444125 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:44.444146 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:44.444543 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:44.444973 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:44.445536 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:44.446956 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:44.447942 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:44.448583 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:44.448735 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:44.451007 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:44.451014 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:44.451030 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:44.451064 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:44.452904 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:44.453369 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:44.453375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:44.453380 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:44.453518 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:44.453799 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:44.453802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:44.453804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:44.454275 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:44.454487 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:44.454725 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 07:05:44.455000 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":103,"threads_started":1}
I20260924 07:05:44.456336 4551 consensus_queue.cc:237] T 00000000000000000000000000000000 P dae868f47c73474fb9e338d3e7bd6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae868f47c73474fb9e338d3e7bd6f5a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33507 } }
I20260924 07:05:44.456344 4546 sys_catalog.cc:564] T 00000000000000000000000000000000 P dae868f47c73474fb9e338d3e7bd6f5a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:05:44.457705 4552 sys_catalog.cc:455] T 00000000000000000000000000000000 P dae868f47c73474fb9e338d3e7bd6f5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dae868f47c73474fb9e338d3e7bd6f5a. Latest consensus state: current_term: 1 leader_uuid: "dae868f47c73474fb9e338d3e7bd6f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae868f47c73474fb9e338d3e7bd6f5a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33507 } } }
I20260924 07:05:44.457824 4552 sys_catalog.cc:458] T 00000000000000000000000000000000 P dae868f47c73474fb9e338d3e7bd6f5a [sys.catalog]: This master's current role is: LEADER
I20260924 07:05:44.458098 4553 sys_catalog.cc:455] T 00000000000000000000000000000000 P dae868f47c73474fb9e338d3e7bd6f5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dae868f47c73474fb9e338d3e7bd6f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae868f47c73474fb9e338d3e7bd6f5a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33507 } } }
I20260924 07:05:44.458679 4553 sys_catalog.cc:458] T 00000000000000000000000000000000 P dae868f47c73474fb9e338d3e7bd6f5a [sys.catalog]: This master's current role is: LEADER
W20260924 07:05:44.460832 4566 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dae868f47c73474fb9e338d3e7bd6f5a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:05:44.460896 4566 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:05:44.460965 4561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:05:44.461613 4561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:05:44.466444 4561 catalog_manager.cc:1347] Generated new cluster ID: 0179efbed8494a818f93f8114705b391
I20260924 07:05:44.466498 4561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:05:44.477396 4561 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:05:44.478562 4561 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:05:44.482553 4561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dae868f47c73474fb9e338d3e7bd6f5a: Generated new TSK 0
I20260924 07:05:44.483203 4561 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
W20260924 07:05:45.284272 3940 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.217.1: real 0.794s user 0.000s sys 0.004s
I20260924 07:05:45.284740 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:33507
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:05:45.401779 4572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:05:45.402029 4572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:05:45.402072 4572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:05:45.406204 4572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:05:45.406266 4572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:05:45.406330 4572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:05:45.411438 4572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:33507
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:05:45.412477 4572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:05:45.413546 4572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:05:45.424628 4578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:45.424659 4577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:45.424608 4580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:05:45.425076 4572 server_base.cc:1034] running on GCE node
I20260924 07:05:45.425524 4572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:05:45.426441 4572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:05:45.427711 4572 hybrid_clock.cc:648] HybridClock initialized: now 1790233545427687 us; error 41 us; skew 500 ppm
Sep 24 07:05:45 dist-test-slave-2p7j krb5kdc[4465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233545, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:05:45.432552 4572 init.cc:377] Logged in from keytab as kudu/127.3.217.1@KRBTEST.COM (short username kudu)
I20260924 07:05:45.434150 4572 webserver.cc:460] Webserver started at http://127.3.217.1:42313/ using document root <none> and password file <none>
I20260924 07:05:45.435067 4572 fs_manager.cc:359] Metadata directory not provided
I20260924 07:05:45.435176 4572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:05:45.435454 4572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:05:45.438635 4572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "78328fd060d4423d9b428b03be85d337"
format_stamp: "Formatted at 2026-09-24 07:05:45 on dist-test-slave-2p7j"
server_key: "90f2934005274541407c635514296663"
server_key_iv: "698a0d9e541d0cd12fcdd6e6679a00a9"
server_key_version: "encryptionkey@0"
I20260924 07:05:45.439388 4572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "78328fd060d4423d9b428b03be85d337"
format_stamp: "Formatted at 2026-09-24 07:05:45 on dist-test-slave-2p7j"
server_key: "90f2934005274541407c635514296663"
server_key_iv: "698a0d9e541d0cd12fcdd6e6679a00a9"
server_key_version: "encryptionkey@0"
I20260924 07:05:45.445147 4572 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 07:05:45.449048 4587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:45.450937 4572 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:05:45.451140 4572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "78328fd060d4423d9b428b03be85d337"
format_stamp: "Formatted at 2026-09-24 07:05:45 on dist-test-slave-2p7j"
server_key: "90f2934005274541407c635514296663"
server_key_iv: "698a0d9e541d0cd12fcdd6e6679a00a9"
server_key_version: "encryptionkey@0"
I20260924 07:05:45.451294 4572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:05:45.486065 4572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:05:45.490379 4572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:05:45.490597 4572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:05:45.491328 4572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:05:45.492451 4572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:05:45.492507 4572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:45.492568 4572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:05:45.492595 4572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:45.503800 4572 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:35667
I20260924 07:05:45.503820 4700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:35667 every 8 connection(s)
I20260924 07:05:45.504976 4572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:05:45.512511 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 4572
I20260924 07:05:45.512642 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:05:45.512918 3940 external_mini_cluster.cc:1428] Setting key bad8b96a2f0d6f6b6a56497f3e034c49
Sep 24 07:05:45 dist-test-slave-2p7j krb5kdc[4465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233545, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:05:45.522119 4703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:45.507146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:40143 (local address 127.3.217.62:33507)
0924 07:05:45.507466 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:45.507470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:45.508313 (+ 843us) server_negotiation.cc:407] Connection header received
0924 07:05:45.510012 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:45.510016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:45.510092 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:45.510186 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:45.511706 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:45.512406 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:45.513912 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:45.514248 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:45.514457 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:45.517144 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:45.517149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:45.517152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:45.517190 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:45.519242 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:45.519869 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:45.519873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:45.519875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:45.519991 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:45.520448 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:45.520452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:45.520454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:45.520734 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:45.520852 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:45.521721 (+ 869us) server_negotiation.cc:299] Negotiation successful
0924 07:05:45.521911 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":116,"threads_started":1}
I20260924 07:05:45.523037 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:45.507533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:33507 (local address 127.3.217.1:40143)
0924 07:05:45.508110 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:45.508166 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:45.509739 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:45.510419 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:45.510433 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:45.510867 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:05:45.511479 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:45.511496 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:45.512588 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:45.512593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:45.513753 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:45.513768 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:45.514507 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:45.514513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:45.514688 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:45.515418 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:05:45.515435 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:05:45.516954 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:05:45.519393 (+ 2439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:45.519400 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:45.519412 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:45.519752 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:45.520123 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:45.520126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:45.520127 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:45.520296 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:45.520854 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:05:45.520861 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:05:45.521316 (+ 455us) client_negotiation.cc:769] Sending connection context
0924 07:05:45.521629 (+ 313us) client_negotiation.cc:240] Negotiation successful
0924 07:05:45.522052 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":146,"threads_started":1}
WARNING: no policy specified for kudu/127.3.217.2@KRBTEST.COM; defaulting to no policy
I20260924 07:05:45.524453 4701 heartbeater.cc:347] Connected to a master server at 127.3.217.62:33507
I20260924 07:05:45.524801 4701 heartbeater.cc:464] Registering TS with master...
I20260924 07:05:45.525460 4701 heartbeater.cc:511] Master 127.3.217.62:33507 requested a full tablet report, sending...
I20260924 07:05:45.527325 4510 ts_manager.cc:194] Registered new tserver with Master: 78328fd060d4423d9b428b03be85d337 (127.3.217.1:35667)
I20260924 07:05:45.528770 4510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.1@KRBTEST.COM'} at 127.3.217.1:40143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.2@KRBTEST.COM" created.
I20260924 07:05:46.531770 4701 heartbeater.cc:503] Master 127.3.217.62:33507 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
W20260924 07:05:46.676478 3940 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.217.2: real 1.161s user 0.000s sys 0.004s
I20260924 07:05:46.677018 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:33507
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:05:46.878985 4707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:05:46.879259 4707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:05:46.879304 4707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:05:46.883246 4707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:05:46.883317 4707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:05:46.883396 4707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:05:46.888541 4707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:33507
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:05:46.889565 4707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:05:46.890573 4707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:05:46.900895 4707 server_base.cc:1034] running on GCE node
W20260924 07:05:46.900856 4713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:46.900861 4715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:46.900874 4712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:05:46.901486 4707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:05:46.902102 4707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:05:46.903280 4707 hybrid_clock.cc:648] HybridClock initialized: now 1790233546903248 us; error 61 us; skew 500 ppm
Sep 24 07:05:46 dist-test-slave-2p7j krb5kdc[4465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233546, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:05:46.915048 4707 init.cc:377] Logged in from keytab as kudu/127.3.217.2@KRBTEST.COM (short username kudu)
I20260924 07:05:46.915984 4707 webserver.cc:460] Webserver started at http://127.3.217.2:44453/ using document root <none> and password file <none>
I20260924 07:05:46.916523 4707 fs_manager.cc:359] Metadata directory not provided
I20260924 07:05:46.916589 4707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:05:46.916775 4707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:05:46.918754 4707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "f554915122c14b90842c7170fcfd0407"
format_stamp: "Formatted at 2026-09-24 07:05:46 on dist-test-slave-2p7j"
server_key: "171e01e12590b6ab151c353573303315"
server_key_iv: "243bb5a051d558a4ae10898344008985"
server_key_version: "encryptionkey@0"
I20260924 07:05:46.919266 4707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "f554915122c14b90842c7170fcfd0407"
format_stamp: "Formatted at 2026-09-24 07:05:46 on dist-test-slave-2p7j"
server_key: "171e01e12590b6ab151c353573303315"
server_key_iv: "243bb5a051d558a4ae10898344008985"
server_key_version: "encryptionkey@0"
I20260924 07:05:46.923480 4707 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:05:46.926278 4722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:46.927695 4707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 07:05:46.927801 4707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "f554915122c14b90842c7170fcfd0407"
format_stamp: "Formatted at 2026-09-24 07:05:46 on dist-test-slave-2p7j"
server_key: "171e01e12590b6ab151c353573303315"
server_key_iv: "243bb5a051d558a4ae10898344008985"
server_key_version: "encryptionkey@0"
I20260924 07:05:46.927894 4707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:05:46.954792 4707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:05:46.958998 4707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:05:46.959178 4707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:05:46.959847 4707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:05:46.960721 4707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:05:46.960772 4707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:46.960824 4707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:05:46.960852 4707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:46.971356 4707 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:43145
I20260924 07:05:46.971596 4835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:43145 every 8 connection(s)
I20260924 07:05:46.972293 4707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:05:46.976890 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 4707
I20260924 07:05:46.977042 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:05:46.977411 3940 external_mini_cluster.cc:1428] Setting key 3d342bcb0fba9c813f361f1f591a193f
Sep 24 07:05:46 dist-test-slave-2p7j krb5kdc[4465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233546, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.217.3@KRBTEST.COM; defaulting to no policy
I20260924 07:05:46.991165 4838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:46.974947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:52621 (local address 127.3.217.62:33507)
0924 07:05:46.975276 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:46.975282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:46.976299 (+ 1017us) server_negotiation.cc:407] Connection header received
0924 07:05:46.978291 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:46.978294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:46.978359 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:46.978450 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:46.981036 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:46.981819 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:46.983453 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:46.983767 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:46.983874 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:46.986713 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:46.986717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:46.986719 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:46.986751 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:46.988591 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:46.989240 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:46.989244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:46.989246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:46.989345 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:46.989712 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:46.989716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:46.989717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:46.989951 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:46.990050 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:46.990775 (+ 725us) server_negotiation.cc:299] Negotiation successful
0924 07:05:46.990949 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":103,"threads_started":1}
I20260924 07:05:46.992317 4839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:46.975452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:33507 (local address 127.3.217.2:52621)
0924 07:05:46.976105 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:46.976165 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:46.977928 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:46.979285 (+ 1357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:46.979299 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:46.979963 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:05:46.980809 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:46.980832 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:46.982013 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:46.982016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:46.983266 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:46.983280 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:46.983956 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:46.983960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:46.984165 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:46.984826 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:05:46.984854 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:05:46.986498 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:05:46.988734 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:46.988742 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:46.988766 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:46.989131 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:46.989449 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:46.989453 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:46.989455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:46.989621 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:46.990064 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:05:46.990073 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:05:46.990482 (+ 409us) client_negotiation.cc:769] Sending connection context
0924 07:05:46.990721 (+ 239us) client_negotiation.cc:240] Negotiation successful
0924 07:05:46.991166 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":161,"threads_started":1}
I20260924 07:05:46.993964 4836 heartbeater.cc:347] Connected to a master server at 127.3.217.62:33507
I20260924 07:05:46.994323 4836 heartbeater.cc:464] Registering TS with master...
I20260924 07:05:46.994930 4836 heartbeater.cc:511] Master 127.3.217.62:33507 requested a full tablet report, sending...
I20260924 07:05:46.996323 4510 ts_manager.cc:194] Registered new tserver with Master: f554915122c14b90842c7170fcfd0407 (127.3.217.2:43145)
I20260924 07:05:46.997145 4510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.2@KRBTEST.COM'} at 127.3.217.2:52621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.3@KRBTEST.COM" created.
I20260924 07:05:47.999570 4836 heartbeater.cc:503] Master 127.3.217.62:33507 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
W20260924 07:05:48.105091 3940 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.217.3: real 1.125s user 0.002s sys 0.002s
I20260924 07:05:48.105603 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:33507
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:05:48.225123 4842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:05:48.225373 4842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:05:48.225421 4842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:05:48.229614 4842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:05:48.229686 4842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:05:48.229751 4842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:05:48.235164 4842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:33507
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:05:48.236172 4842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:05:48.237516 4842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:05:48.249105 4847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:48.249105 4850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:48.249476 4848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:05:48.250178 4842 server_base.cc:1034] running on GCE node
I20260924 07:05:48.250643 4842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:05:48.251827 4842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:05:48.255278 4842 hybrid_clock.cc:648] HybridClock initialized: now 1790233548255246 us; error 65 us; skew 500 ppm
Sep 24 07:05:48 dist-test-slave-2p7j krb5kdc[4465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233548, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:05:48.258546 4842 init.cc:377] Logged in from keytab as kudu/127.3.217.3@KRBTEST.COM (short username kudu)
I20260924 07:05:48.259480 4842 webserver.cc:460] Webserver started at http://127.3.217.3:43607/ using document root <none> and password file <none>
I20260924 07:05:48.260250 4842 fs_manager.cc:359] Metadata directory not provided
I20260924 07:05:48.260334 4842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:05:48.260533 4842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:05:48.262929 4842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "b1adb86cc18f4466b190a3f1502a8d49"
format_stamp: "Formatted at 2026-09-24 07:05:48 on dist-test-slave-2p7j"
server_key: "d971ca789cc7285bc4574a4dc6e4c509"
server_key_iv: "ebaa391fc71ea040fffea7e19795b235"
server_key_version: "encryptionkey@0"
I20260924 07:05:48.263619 4842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "b1adb86cc18f4466b190a3f1502a8d49"
format_stamp: "Formatted at 2026-09-24 07:05:48 on dist-test-slave-2p7j"
server_key: "d971ca789cc7285bc4574a4dc6e4c509"
server_key_iv: "ebaa391fc71ea040fffea7e19795b235"
server_key_version: "encryptionkey@0"
I20260924 07:05:48.268399 4842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260924 07:05:48.271781 4857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:48.274721 4842 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260924 07:05:48.275100 4842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "b1adb86cc18f4466b190a3f1502a8d49"
format_stamp: "Formatted at 2026-09-24 07:05:48 on dist-test-slave-2p7j"
server_key: "d971ca789cc7285bc4574a4dc6e4c509"
server_key_iv: "ebaa391fc71ea040fffea7e19795b235"
server_key_version: "encryptionkey@0"
I20260924 07:05:48.275409 4842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:05:48.305120 4842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:05:48.311731 4842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:05:48.311910 4842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:05:48.312709 4842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:05:48.313886 4842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:05:48.313935 4842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:48.313990 4842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:05:48.314018 4842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:48.324182 4842 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:36249
I20260924 07:05:48.324234 4970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:36249 every 8 connection(s)
I20260924 07:05:48.325564 4842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:05:48.333132 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 4842
I20260924 07:05:48.333282 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:05:48.333642 3940 external_mini_cluster.cc:1428] Setting key f35be052b6ed0271ee7d6067ecceef23
Sep 24 07:05:48 dist-test-slave-2p7j krb5kdc[4465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233548, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:05:48.343441 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:48.328279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:50449 (local address 127.3.217.62:33507)
0924 07:05:48.328654 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:48.328660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:48.330353 (+ 1693us) server_negotiation.cc:407] Connection header received
0924 07:05:48.331926 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:48.331931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:48.331989 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:48.332075 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:48.333550 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.334361 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.336078 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.336352 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.336462 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:48.339172 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:48.339175 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:48.339177 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:48.339201 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:48.341150 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:48.341727 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:48.341730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:48.341732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:48.341834 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:48.342260 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:48.342263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:48.342264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:48.342459 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:48.342542 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:48.343151 (+ 609us) server_negotiation.cc:299] Negotiation successful
0924 07:05:48.343271 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":94,"threads_started":1}
I20260924 07:05:48.344357 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:48.329013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:33507 (local address 127.3.217.3:50449)
0924 07:05:48.330003 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:48.330050 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:48.331631 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:48.332277 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:48.332284 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:48.332698 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:05:48.333226 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:48.333239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.334844 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.334847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:48.335862 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:48.335871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.336508 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.336512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:48.336676 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:48.337325 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:05:48.337341 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:05:48.338991 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:05:48.341329 (+ 2338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:48.341334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:48.341347 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:48.341592 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:48.341951 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:48.341954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:48.341955 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:48.342091 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:48.342565 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:05:48.342570 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:05:48.342837 (+ 267us) client_negotiation.cc:769] Sending connection context
0924 07:05:48.343024 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 07:05:48.343307 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":118,"threads_started":1}
I20260924 07:05:48.345749 4971 heartbeater.cc:347] Connected to a master server at 127.3.217.62:33507
I20260924 07:05:48.346031 4971 heartbeater.cc:464] Registering TS with master...
I20260924 07:05:48.346517 4971 heartbeater.cc:511] Master 127.3.217.62:33507 requested a full tablet report, sending...
I20260924 07:05:48.347635 4510 ts_manager.cc:194] Registered new tserver with Master: b1adb86cc18f4466b190a3f1502a8d49 (127.3.217.3:36249)
I20260924 07:05:48.348222 4510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.3@KRBTEST.COM'} at 127.3.217.3:50449
I20260924 07:05:48.349293 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:05:48.364218 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:48.353897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52838 (local address 127.3.217.62:33507)
0924 07:05:48.354319 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:48.354323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:48.354507 (+ 184us) server_negotiation.cc:407] Connection header received
0924 07:05:48.354580 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:48.354582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:48.354623 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:48.354689 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:48.355673 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.356534 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.357874 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.358192 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.358335 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:48.359980 (+ 1645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:48.359983 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:48.359985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:48.360019 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:48.361916 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:48.362463 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:48.362466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:48.362467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:48.362563 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:48.363074 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:48.363077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:48.363078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:48.363322 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:48.363459 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:48.363961 (+ 502us) server_negotiation.cc:299] Negotiation successful
0924 07:05:48.364080 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309}
I20260924 07:05:48.369805 4510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:05:48.372943 4510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:05:48.391047 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:48.384106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:43145 (local address 127.0.0.1:51102)
0924 07:05:48.384990 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:48.385048 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:48.385189 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:48.386738 (+ 1549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:48.386741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:48.386758 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:05:48.386828 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:48.386833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.388075 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.388078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:48.389391 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:48.389401 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.390527 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.390531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:48.390632 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:48.390677 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 07:05:48.390784 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 07:05:48.390898 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"mutex_wait_us":218,"thread_start_us":127,"threads_started":1}
I20260924 07:05:48.391041 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:48.384826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51102 (local address 127.3.217.2:43145)
0924 07:05:48.386293 (+ 1467us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:48.386298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:48.386319 (+ 21us) server_negotiation.cc:407] Connection header received
0924 07:05:48.386395 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:48.386400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:48.386572 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:48.386760 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:05:48.386917 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.387967 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.389528 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.390400 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.390507 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:48.390659 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:48.390808 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 07:05:48.390888 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1292,"thread_start_us":102,"threads_started":1}
I20260924 07:05:48.391197 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:48.384173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:36249 (local address 127.0.0.1:34726)
0924 07:05:48.385184 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:48.385198 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:48.385293 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:48.386047 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:48.386053 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:48.386077 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:05:48.386297 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:48.386305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.387885 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.387889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:48.389899 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:48.389907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.390865 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.390869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:48.390948 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:48.390969 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:05:48.391030 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 07:05:48.391089 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"mutex_wait_us":943,"thread_start_us":81,"threads_started":1}
I20260924 07:05:48.391542 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:48.385212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.3.217.3:36249)
0924 07:05:48.385528 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:48.385539 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:48.385562 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:05:48.385637 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:48.385642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:48.385879 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:48.386039 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:05:48.386439 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.387620 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.390010 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.390576 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.390669 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:48.390808 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:48.391281 (+ 473us) server_negotiation.cc:299] Negotiation successful
0924 07:05:48.391380 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":99,"threads_started":1}
I20260924 07:05:48.392185 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:48.384845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:35667 (local address 127.0.0.1:46382)
0924 07:05:48.385893 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:48.385906 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:48.385999 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:48.386455 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:48.386458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:48.386474 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:05:48.386555 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:48.386561 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.388024 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.388027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:48.389450 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:48.389461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.391812 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.391816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:48.391900 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:48.391923 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:05:48.391995 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 07:05:48.392062 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"mutex_wait_us":77,"thread_start_us":84,"threads_started":1}
I20260924 07:05:48.392292 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:48.385582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46382 (local address 127.3.217.1:35667)
0924 07:05:48.385850 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:48.385855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:48.386014 (+ 159us) server_negotiation.cc:407] Connection header received
0924 07:05:48.386165 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:48.386172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:48.386355 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:48.386482 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:05:48.386684 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.387897 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.389552 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.390400 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.390507 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:48.390649 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:48.392041 (+ 1392us) server_negotiation.cc:299] Negotiation successful
0924 07:05:48.392161 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":78,"threads_started":1}
I20260924 07:05:48.394022 4905 tablet_service.cc:1467] Processing CreateTablet for tablet 79fa1a4e80b8457e868dfd7f011bed5f (DEFAULT_TABLE table=test-table [id=0630dca591124c2d8329d05ee9fcd416]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:05:48.394024 4770 tablet_service.cc:1467] Processing CreateTablet for tablet 79fa1a4e80b8457e868dfd7f011bed5f (DEFAULT_TABLE table=test-table [id=0630dca591124c2d8329d05ee9fcd416]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:05:48.394943 4905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79fa1a4e80b8457e868dfd7f011bed5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:48.394944 4770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79fa1a4e80b8457e868dfd7f011bed5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:48.397987 4635 tablet_service.cc:1467] Processing CreateTablet for tablet 79fa1a4e80b8457e868dfd7f011bed5f (DEFAULT_TABLE table=test-table [id=0630dca591124c2d8329d05ee9fcd416]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:05:48.399003 4635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79fa1a4e80b8457e868dfd7f011bed5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:48.401286 4990 tablet_bootstrap.cc:492] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49: Bootstrap starting.
I20260924 07:05:48.402725 4991 tablet_bootstrap.cc:492] T 79fa1a4e80b8457e868dfd7f011bed5f P f554915122c14b90842c7170fcfd0407: Bootstrap starting.
I20260924 07:05:48.403415 4990 tablet_bootstrap.cc:654] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:48.404610 4990 log.cc:824] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49: Log is configured to *not* fsync() on all Append() calls
I20260924 07:05:48.405232 4991 tablet_bootstrap.cc:654] T 79fa1a4e80b8457e868dfd7f011bed5f P f554915122c14b90842c7170fcfd0407: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:48.405990 4991 log.cc:824] T 79fa1a4e80b8457e868dfd7f011bed5f P f554915122c14b90842c7170fcfd0407: Log is configured to *not* fsync() on all Append() calls
I20260924 07:05:48.406605 4993 tablet_bootstrap.cc:492] T 79fa1a4e80b8457e868dfd7f011bed5f P 78328fd060d4423d9b428b03be85d337: Bootstrap starting.
I20260924 07:05:48.406801 4990 tablet_bootstrap.cc:492] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49: No bootstrap required, opened a new log
I20260924 07:05:48.406977 4990 ts_tablet_manager.cc:1397] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:05:48.408854 4991 tablet_bootstrap.cc:492] T 79fa1a4e80b8457e868dfd7f011bed5f P f554915122c14b90842c7170fcfd0407: No bootstrap required, opened a new log
I20260924 07:05:48.409332 4991 ts_tablet_manager.cc:1397] T 79fa1a4e80b8457e868dfd7f011bed5f P f554915122c14b90842c7170fcfd0407: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20260924 07:05:48.409909 4990 raft_consensus.cc:348] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f554915122c14b90842c7170fcfd0407" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43145 } } peers { permanent_uuid: "b1adb86cc18f4466b190a3f1502a8d49" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36249 } } peers { permanent_uuid: "78328fd060d4423d9b428b03be85d337" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35667 } }
I20260924 07:05:48.410120 4990 raft_consensus.cc:374] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:48.410152 4990 raft_consensus.cc:729] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1adb86cc18f4466b190a3f1502a8d49, State: Initialized, Role: FOLLOWER
I20260924 07:05:48.410153 4993 tablet_bootstrap.cc:654] T 79fa1a4e80b8457e868dfd7f011bed5f P 78328fd060d4423d9b428b03be85d337: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:48.410530 4990 consensus_queue.cc:260] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f554915122c14b90842c7170fcfd0407" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43145 } } peers { permanent_uuid: "b1adb86cc18f4466b190a3f1502a8d49" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36249 } } peers { permanent_uuid: "78328fd060d4423d9b428b03be85d337" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35667 } }
I20260924 07:05:48.411073 4993 log.cc:824] T 79fa1a4e80b8457e868dfd7f011bed5f P 78328fd060d4423d9b428b03be85d337: Log is configured to *not* fsync() on all Append() calls
I20260924 07:05:48.411095 4971 heartbeater.cc:503] Master 127.3.217.62:33507 was elected leader, sending a full tablet report...
I20260924 07:05:48.411432 4990 ts_tablet_manager.cc:1428] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260924 07:05:48.413689 4993 tablet_bootstrap.cc:492] T 79fa1a4e80b8457e868dfd7f011bed5f P 78328fd060d4423d9b428b03be85d337: No bootstrap required, opened a new log
I20260924 07:05:48.413988 4993 ts_tablet_manager.cc:1397] T 79fa1a4e80b8457e868dfd7f011bed5f P 78328fd060d4423d9b428b03be85d337: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 07:05:48.414520 4991 raft_consensus.cc:348] T 79fa1a4e80b8457e868dfd7f011bed5f P f554915122c14b90842c7170fcfd0407 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f554915122c14b90842c7170fcfd0407" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43145 } } peers { permanent_uuid: "b1adb86cc18f4466b190a3f1502a8d49" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36249 } } peers { permanent_uuid: "78328fd060d4423d9b428b03be85d337" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35667 } }
I20260924 07:05:48.414857 4991 raft_consensus.cc:374] T 79fa1a4e80b8457e868dfd7f011bed5f P f554915122c14b90842c7170fcfd0407 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:48.414908 4991 raft_consensus.cc:729] T 79fa1a4e80b8457e868dfd7f011bed5f P f554915122c14b90842c7170fcfd0407 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f554915122c14b90842c7170fcfd0407, State: Initialized, Role: FOLLOWER
I20260924 07:05:48.415613 4991 consensus_queue.cc:260] T 79fa1a4e80b8457e868dfd7f011bed5f P f554915122c14b90842c7170fcfd0407 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f554915122c14b90842c7170fcfd0407" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43145 } } peers { permanent_uuid: "b1adb86cc18f4466b190a3f1502a8d49" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36249 } } peers { permanent_uuid: "78328fd060d4423d9b428b03be85d337" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35667 } }
I20260924 07:05:48.416823 4991 ts_tablet_manager.cc:1428] T 79fa1a4e80b8457e868dfd7f011bed5f P f554915122c14b90842c7170fcfd0407: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260924 07:05:48.418527 4993 raft_consensus.cc:348] T 79fa1a4e80b8457e868dfd7f011bed5f P 78328fd060d4423d9b428b03be85d337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f554915122c14b90842c7170fcfd0407" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43145 } } peers { permanent_uuid: "b1adb86cc18f4466b190a3f1502a8d49" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36249 } } peers { permanent_uuid: "78328fd060d4423d9b428b03be85d337" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35667 } }
I20260924 07:05:48.418774 4993 raft_consensus.cc:374] T 79fa1a4e80b8457e868dfd7f011bed5f P 78328fd060d4423d9b428b03be85d337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:48.418831 4993 raft_consensus.cc:729] T 79fa1a4e80b8457e868dfd7f011bed5f P 78328fd060d4423d9b428b03be85d337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78328fd060d4423d9b428b03be85d337, State: Initialized, Role: FOLLOWER
I20260924 07:05:48.419291 4993 consensus_queue.cc:260] T 79fa1a4e80b8457e868dfd7f011bed5f P 78328fd060d4423d9b428b03be85d337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f554915122c14b90842c7170fcfd0407" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43145 } } peers { permanent_uuid: "b1adb86cc18f4466b190a3f1502a8d49" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36249 } } peers { permanent_uuid: "78328fd060d4423d9b428b03be85d337" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35667 } }
I20260924 07:05:48.420238 4993 ts_tablet_manager.cc:1428] T 79fa1a4e80b8457e868dfd7f011bed5f P 78328fd060d4423d9b428b03be85d337: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
W20260924 07:05:48.474365 4837 tablet.cc:2369] T 79fa1a4e80b8457e868dfd7f011bed5f P f554915122c14b90842c7170fcfd0407: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:05:48.502271 4995 raft_consensus.cc:482] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:05:48.502558 4995 raft_consensus.cc:504] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f554915122c14b90842c7170fcfd0407" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43145 } } peers { permanent_uuid: "b1adb86cc18f4466b190a3f1502a8d49" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36249 } } peers { permanent_uuid: "78328fd060d4423d9b428b03be85d337" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35667 } }
I20260924 07:05:48.504420 4995 leader_election.cc:288] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f554915122c14b90842c7170fcfd0407 (127.3.217.2:43145), 78328fd060d4423d9b428b03be85d337 (127.3.217.1:35667)
W20260924 07:05:48.508514 4702 tablet.cc:2369] T 79fa1a4e80b8457e868dfd7f011bed5f P 78328fd060d4423d9b428b03be85d337: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:05:48.509441 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:48.505130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:47313 (local address 127.3.217.2:43145)
0924 07:05:48.505250 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:48.505253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:48.505550 (+ 297us) server_negotiation.cc:407] Connection header received
0924 07:05:48.505783 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:48.505786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:48.505838 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:48.505919 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:05:48.506132 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.506631 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.508124 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.508717 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.508848 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:48.508914 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:48.509237 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 07:05:48.509322 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 07:05:48.509449 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:48.504754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:35667 (local address 127.3.217.3:50669)
0924 07:05:48.505065 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:48.505150 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:48.505283 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:48.505561 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:48.505564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:48.505582 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:05:48.505709 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:48.505715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.506714 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.506717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:48.507491 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:48.507525 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.509086 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.509089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:48.509192 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:48.509210 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 07:05:48.509263 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 07:05:48.509315 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":3,"spinlock_wait_cycles":23296}
I20260924 07:05:48.509456 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:48.504754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:43145 (local address 127.3.217.3:47313)
0924 07:05:48.505463 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:48.505477 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:48.505635 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:48.505923 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:48.505926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:48.505940 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:05:48.506040 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:48.506045 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.506757 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.506760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:48.507982 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:48.507991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.508999 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.509003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:48.509094 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:48.509119 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 07:05:48.509203 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 07:05:48.509275 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":153,"spinlock_wait_cycles":768,"thread_start_us":262,"threads_started":1}
I20260924 07:05:48.510126 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:48.504843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:50669 (local address 127.3.217.1:35667)
0924 07:05:48.504995 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:48.505000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:48.505373 (+ 373us) server_negotiation.cc:407] Connection header received
0924 07:05:48.505423 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:48.505425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:48.505466 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:48.505525 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:05:48.505823 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.506599 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.507624 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.508905 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.508991 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:48.509040 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:48.509957 (+ 917us) server_negotiation.cc:299] Negotiation successful
0924 07:05:48.510046 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 07:05:48.510339 4790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79fa1a4e80b8457e868dfd7f011bed5f" candidate_uuid: "b1adb86cc18f4466b190a3f1502a8d49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f554915122c14b90842c7170fcfd0407" is_pre_election: true
I20260924 07:05:48.510582 4655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79fa1a4e80b8457e868dfd7f011bed5f" candidate_uuid: "b1adb86cc18f4466b190a3f1502a8d49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78328fd060d4423d9b428b03be85d337" is_pre_election: true
I20260924 07:05:48.510655 4790 raft_consensus.cc:2454] T 79fa1a4e80b8457e868dfd7f011bed5f P f554915122c14b90842c7170fcfd0407 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1adb86cc18f4466b190a3f1502a8d49 in term 0.
I20260924 07:05:48.510829 4655 raft_consensus.cc:2454] T 79fa1a4e80b8457e868dfd7f011bed5f P 78328fd060d4423d9b428b03be85d337 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1adb86cc18f4466b190a3f1502a8d49 in term 0.
I20260924 07:05:48.511138 4859 leader_election.cc:302] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1adb86cc18f4466b190a3f1502a8d49, f554915122c14b90842c7170fcfd0407; no voters:
I20260924 07:05:48.511412 4995 raft_consensus.cc:2781] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:05:48.511485 4995 raft_consensus.cc:482] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:05:48.511554 4995 raft_consensus.cc:3037] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:48.573160 4995 raft_consensus.cc:504] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f554915122c14b90842c7170fcfd0407" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43145 } } peers { permanent_uuid: "b1adb86cc18f4466b190a3f1502a8d49" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36249 } } peers { permanent_uuid: "78328fd060d4423d9b428b03be85d337" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35667 } }
I20260924 07:05:48.573614 4995 leader_election.cc:288] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49 [CANDIDATE]: Term 1 election: Requested vote from peers f554915122c14b90842c7170fcfd0407 (127.3.217.2:43145), 78328fd060d4423d9b428b03be85d337 (127.3.217.1:35667)
I20260924 07:05:48.574136 4790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79fa1a4e80b8457e868dfd7f011bed5f" candidate_uuid: "b1adb86cc18f4466b190a3f1502a8d49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f554915122c14b90842c7170fcfd0407"
I20260924 07:05:48.574285 4790 raft_consensus.cc:3037] T 79fa1a4e80b8457e868dfd7f011bed5f P f554915122c14b90842c7170fcfd0407 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:48.574395 4655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79fa1a4e80b8457e868dfd7f011bed5f" candidate_uuid: "b1adb86cc18f4466b190a3f1502a8d49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78328fd060d4423d9b428b03be85d337"
I20260924 07:05:48.574630 4655 raft_consensus.cc:3037] T 79fa1a4e80b8457e868dfd7f011bed5f P 78328fd060d4423d9b428b03be85d337 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:48.575763 4790 raft_consensus.cc:2454] T 79fa1a4e80b8457e868dfd7f011bed5f P f554915122c14b90842c7170fcfd0407 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1adb86cc18f4466b190a3f1502a8d49 in term 1.
I20260924 07:05:48.576220 4655 raft_consensus.cc:2454] T 79fa1a4e80b8457e868dfd7f011bed5f P 78328fd060d4423d9b428b03be85d337 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1adb86cc18f4466b190a3f1502a8d49 in term 1.
I20260924 07:05:48.576236 4859 leader_election.cc:302] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1adb86cc18f4466b190a3f1502a8d49, f554915122c14b90842c7170fcfd0407; no voters:
I20260924 07:05:48.576426 4995 raft_consensus.cc:2781] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:05:48.576678 4995 raft_consensus.cc:686] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49 [term 1 LEADER]: Becoming Leader. State: Replica: b1adb86cc18f4466b190a3f1502a8d49, State: Running, Role: LEADER
I20260924 07:05:48.577073 4995 consensus_queue.cc:237] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f554915122c14b90842c7170fcfd0407" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43145 } } peers { permanent_uuid: "b1adb86cc18f4466b190a3f1502a8d49" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36249 } } peers { permanent_uuid: "78328fd060d4423d9b428b03be85d337" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35667 } }
W20260924 07:05:48.577250 4972 tablet.cc:2369] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:05:48.580598 4510 catalog_manager.cc:5495] T 79fa1a4e80b8457e868dfd7f011bed5f P b1adb86cc18f4466b190a3f1502a8d49 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1adb86cc18f4466b190a3f1502a8d49 (127.3.217.3). New cstate: current_term: 1 leader_uuid: "b1adb86cc18f4466b190a3f1502a8d49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f554915122c14b90842c7170fcfd0407" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1adb86cc18f4466b190a3f1502a8d49" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78328fd060d4423d9b428b03be85d337" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35667 } health_report { overall_health: UNKNOWN } } }
Sep 24 07:05:48 dist-test-slave-2p7j krb5kdc[4465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.1@KRBTEST.COM
I20260924 07:05:48.628831 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:48.618341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46392 (local address 127.3.217.1:35667)
0924 07:05:48.618485 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:48.618490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:48.618623 (+ 133us) server_negotiation.cc:407] Connection header received
0924 07:05:48.618761 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:48.618765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:48.618808 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:48.618873 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:48.619577 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.620091 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.621063 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.621382 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.621484 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:48.624361 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:48.624373 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:48.624389 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:48.624410 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:48.626432 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:48.626935 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:48.626945 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:48.626947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:48.627170 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:48.627592 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:48.627595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:48.627597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:48.627935 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:48.628209 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:48.628494 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 07:05:48.628644 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 07:05:48 dist-test-slave-2p7j krb5kdc[4465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233548, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 07:05:48 dist-test-slave-2p7j krb5kdc[4465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233548, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.217.1@KRBTEST.COM
I20260924 07:05:48.666302 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:48.654048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46400 (local address 127.3.217.1:35667)
0924 07:05:48.657039 (+ 2991us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:48.657043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:48.657059 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:05:48.657109 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:48.657112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:48.657157 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:48.657276 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:48.657976 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.658525 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.659711 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.660030 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.660173 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:48.663037 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:48.663040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:48.663042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:48.663066 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:48.664386 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:48.665015 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:48.665018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:48.665019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:48.665089 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:48.665414 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:48.665417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:48.665418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:48.665600 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:48.665775 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:48.666042 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 07:05:48.666168 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2901}
Sep 24 07:05:48 dist-test-slave-2p7j krb5kdc[4465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 07:05:48 dist-test-slave-2p7j krb5kdc[4465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.1@KRBTEST.COM
I20260924 07:05:48.766841 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:48.758165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46402 (local address 127.3.217.1:35667)
0924 07:05:48.758344 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:48.758349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:48.758466 (+ 117us) server_negotiation.cc:407] Connection header received
0924 07:05:48.758622 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:48.758626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:48.758675 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:48.758746 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:48.759563 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.760109 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.761503 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:48.761721 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:48.761812 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:48.764264 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:48.764268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:48.764271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:48.764293 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:48.765455 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:48.765810 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:48.765814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:48.765816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:48.765882 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:48.766112 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:48.766115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:48.766117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:48.766265 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:48.766389 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:48.766577 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 07:05:48.766690 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 07:05:48.768193 4615 tablet_service.cc:2908] Rejecting scan request for tablet 79fa1a4e80b8457e868dfd7f011bed5f: Uninitialized: clean time has not yet been initialized
I20260924 07:05:48.770195 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 4572
I20260924 07:05:48.775681 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 4707
I20260924 07:05:48.781035 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 4842
I20260924 07:05:48.785563 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 4478
2026-09-24T07:05:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5598 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:05:50 dist-test-slave-2p7j krb5kdc[5014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:05:50 dist-test-slave-2p7j krb5kdc[5014](info): set up 2 sockets
Sep 24 07:05:50 dist-test-slave-2p7j krb5kdc[5014](info): commencing operation
krb5kdc: starting...
W20260924 07:05:50.967730 3940 mini_kdc.cc:121] Time spent starting KDC: real 2.160s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:05:51 dist-test-slave-2p7j krb5kdc[5014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:05:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:05:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:05:51.090873 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:36735
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:44097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:36735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:05:51.197868 5027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:05:51.198117 5027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:05:51.198172 5027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:05:51.202121 5027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:05:51.202172 5027 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 07:05:51.202195 5027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:05:51.202219 5027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:05:51.202248 5027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:05:51.202271 5027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:05:51.207419 5027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:44097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:36735
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:36735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:05:51.208359 5027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:05:51.209267 5027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:05:51.218581 5033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:51.218591 5035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:51.218850 5032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:05:51.218933 5027 server_base.cc:1034] running on GCE node
I20260924 07:05:51.219440 5027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:05:51.220534 5027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:05:51.221788 5027 hybrid_clock.cc:648] HybridClock initialized: now 1790233551221754 us; error 31 us; skew 500 ppm
Sep 24 07:05:51 dist-test-slave-2p7j krb5kdc[5014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233551, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:05:51.225057 5027 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:05:51.225932 5027 webserver.cc:460] Webserver started at http://127.3.217.62:45227/ using document root <none> and password file <none>
I20260924 07:05:51.226451 5027 fs_manager.cc:359] Metadata directory not provided
I20260924 07:05:51.226517 5027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:05:51.226691 5027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:05:51.228626 5027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "7c91b156a1f149a08cce4ab8c45e1753"
format_stamp: "Formatted at 2026-09-24 07:05:51 on dist-test-slave-2p7j"
server_key: "770da79a8dd47fbef5a26e1bd7543a8d"
server_key_iv: "9929c15bf17ae6d070e49f5da125d1f6"
server_key_version: "encryptionkey@0"
I20260924 07:05:51.229143 5027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "7c91b156a1f149a08cce4ab8c45e1753"
format_stamp: "Formatted at 2026-09-24 07:05:51 on dist-test-slave-2p7j"
server_key: "770da79a8dd47fbef5a26e1bd7543a8d"
server_key_iv: "9929c15bf17ae6d070e49f5da125d1f6"
server_key_version: "encryptionkey@0"
I20260924 07:05:51.233098 5027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:05:51.235797 5042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:51.237484 5027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:05:51.237617 5027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "7c91b156a1f149a08cce4ab8c45e1753"
format_stamp: "Formatted at 2026-09-24 07:05:51 on dist-test-slave-2p7j"
server_key: "770da79a8dd47fbef5a26e1bd7543a8d"
server_key_iv: "9929c15bf17ae6d070e49f5da125d1f6"
server_key_version: "encryptionkey@0"
I20260924 07:05:51.237710 5027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:05:51.252835 5027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:05:51.256683 5027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:05:51.256899 5027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:05:51.264688 5027 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:36735
I20260924 07:05:51.264739 5104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:36735 every 8 connection(s)
I20260924 07:05:51.265606 5027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:05:51.266583 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 5027
I20260924 07:05:51.266691 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:05:51.266980 3940 external_mini_cluster.cc:1428] Setting key 5d278db0a7fe5594df884431fd7e10a7
I20260924 07:05:51.268851 5105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:51.274688 5105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753: Bootstrap starting.
Sep 24 07:05:51 dist-test-slave-2p7j krb5kdc[5014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:05:51.277176 5105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:51.277899 5105 log.cc:824] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753: Log is configured to *not* fsync() on all Append() calls
I20260924 07:05:51.279862 5105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753: No bootstrap required, opened a new log
I20260924 07:05:51.282656 5108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:51.268689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59394 (local address 127.3.217.62:36735)
0924 07:05:51.269315 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:51.269335 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:51.269385 (+ 50us) server_negotiation.cc:407] Connection header received
0924 07:05:51.270177 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:51.270198 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:51.270623 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:51.270990 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:51.271862 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.272871 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.274107 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.274754 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.275002 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:51.277720 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:51.277727 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:51.277744 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:51.277780 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:51.279795 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:51.280290 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:51.280295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:51.280299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:51.280410 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:51.280780 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:51.280783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:51.280784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:51.281168 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:51.281335 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:51.281659 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 07:05:51.281900 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":214,"threads_started":1}
I20260924 07:05:51.283841 5105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c91b156a1f149a08cce4ab8c45e1753" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36735 } }
I20260924 07:05:51.284163 5105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:51.284232 5105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c91b156a1f149a08cce4ab8c45e1753, State: Initialized, Role: FOLLOWER
I20260924 07:05:51.284669 5105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c91b156a1f149a08cce4ab8c45e1753" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36735 } }
I20260924 07:05:51.284823 5105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:05:51.284902 5105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:05:51.284997 5105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:51.285820 5105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c91b156a1f149a08cce4ab8c45e1753" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36735 } }
I20260924 07:05:51.286132 5105 leader_election.cc:302] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c91b156a1f149a08cce4ab8c45e1753; no voters:
I20260924 07:05:51.286415 5105 leader_election.cc:288] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:05:51.286760 5110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:05:51.287047 5110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753 [term 1 LEADER]: Becoming Leader. State: Replica: 7c91b156a1f149a08cce4ab8c45e1753, State: Running, Role: LEADER
I20260924 07:05:51.287595 5110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c91b156a1f149a08cce4ab8c45e1753" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36735 } }
I20260924 07:05:51.287674 5105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:05:51.289746 5111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c91b156a1f149a08cce4ab8c45e1753" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c91b156a1f149a08cce4ab8c45e1753" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36735 } } }
I20260924 07:05:51.289872 5111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753 [sys.catalog]: This master's current role is: LEADER
I20260924 07:05:51.290259 5112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c91b156a1f149a08cce4ab8c45e1753. Latest consensus state: current_term: 1 leader_uuid: "7c91b156a1f149a08cce4ab8c45e1753" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c91b156a1f149a08cce4ab8c45e1753" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36735 } } }
I20260924 07:05:51.290493 5112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753 [sys.catalog]: This master's current role is: LEADER
W20260924 07:05:51.293283 5124 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:05:51.293509 5124 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:05:51.293614 5125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:05:51.294471 5125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:05:51.299580 5125 catalog_manager.cc:1347] Generated new cluster ID: feede3bc765c465f9b2f6a92e24b623b
I20260924 07:05:51.299631 5125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:05:51.308888 5125 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:05:51.310168 5125 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:05:51.316093 5125 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753: Generated new TSK 0
I20260924 07:05:51.317164 5125 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
I20260924 07:05:51.356164 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:36735
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:44097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:05:51.461728 5131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:05:51.461938 5131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:05:51.461978 5131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:05:51.465854 5131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:05:51.465903 5131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:05:51.465938 5131 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:05:51.465974 5131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:05:51.471141 5131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:44097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.217.62:36735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:05:51.472064 5131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:05:51.472968 5131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:05:51.481750 5136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:51.481980 5139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:51.482038 5137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:05:51.482280 5131 server_base.cc:1034] running on GCE node
I20260924 07:05:51.482632 5131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:05:51.483275 5131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:05:51.484453 5131 hybrid_clock.cc:648] HybridClock initialized: now 1790233551484373 us; error 91 us; skew 500 ppm
Sep 24 07:05:51 dist-test-slave-2p7j krb5kdc[5014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233551, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:05:51.487128 5131 init.cc:377] Logged in from keytab as kudu/127.3.217.1@KRBTEST.COM (short username kudu)
I20260924 07:05:51.488358 5131 webserver.cc:460] Webserver started at http://127.3.217.1:39269/ using document root <none> and password file <none>
I20260924 07:05:51.489362 5131 fs_manager.cc:359] Metadata directory not provided
I20260924 07:05:51.489475 5131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:05:51.489895 5131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:05:51.491851 5131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "4bbe530d020145e1b47b5e84f0ce5f39"
format_stamp: "Formatted at 2026-09-24 07:05:51 on dist-test-slave-2p7j"
server_key: "093d0761dc18621142da61451dc85467"
server_key_iv: "f8fb52376e3d0247b563394f0178b4a1"
server_key_version: "encryptionkey@0"
I20260924 07:05:51.492340 5131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "4bbe530d020145e1b47b5e84f0ce5f39"
format_stamp: "Formatted at 2026-09-24 07:05:51 on dist-test-slave-2p7j"
server_key: "093d0761dc18621142da61451dc85467"
server_key_iv: "f8fb52376e3d0247b563394f0178b4a1"
server_key_version: "encryptionkey@0"
I20260924 07:05:51.495755 5131 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:05:51.498046 5146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:51.499185 5131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:05:51.499290 5131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "4bbe530d020145e1b47b5e84f0ce5f39"
format_stamp: "Formatted at 2026-09-24 07:05:51 on dist-test-slave-2p7j"
server_key: "093d0761dc18621142da61451dc85467"
server_key_iv: "f8fb52376e3d0247b563394f0178b4a1"
server_key_version: "encryptionkey@0"
I20260924 07:05:51.499380 5131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:05:51.514984 5131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:05:51.517683 5131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:05:51.517851 5131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:05:51.519137 5131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:05:51.519184 5131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:51.519233 5131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:05:51.519264 5131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:05:51 dist-test-slave-2p7j krb5kdc[5014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233551, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:05:51.529493 5131 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:35761
I20260924 07:05:51.529616 5261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:35761 every 8 connection(s)
I20260924 07:05:51.530298 5131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:05:51.531458 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 5131
I20260924 07:05:51.531601 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:05:51.531910 3940 external_mini_cluster.cc:1428] Setting key 23172d4bf632483b68f04b6f37e27e4d
I20260924 07:05:51.532315 5108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:51.519796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:45531 (local address 127.3.217.62:36735)
0924 07:05:51.519910 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:51.519914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:51.521471 (+ 1557us) server_negotiation.cc:407] Connection header received
0924 07:05:51.522655 (+ 1184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:51.522659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:51.522713 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:51.522796 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:51.523919 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.524583 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.525737 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.525986 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.526084 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:51.528553 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:51.528557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:51.528559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:51.528582 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:51.530016 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:51.530498 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:51.530502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:51.530504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:51.530593 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:51.530986 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:51.530988 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:51.530990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:51.531279 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:51.531365 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:51.532063 (+ 698us) server_negotiation.cc:299] Negotiation successful
0924 07:05:51.532175 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:05:51.533113 5155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:51.520784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36735 (local address 127.3.217.1:45531)
0924 07:05:51.521334 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:51.521374 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:51.522409 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:51.522929 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:51.522935 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:51.523256 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:05:51.523777 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:51.523791 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.524703 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.524707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:51.525609 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:51.525629 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.526146 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.526150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:51.526318 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:51.526829 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:05:51.526846 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:05:51.528392 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:05:51.530133 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:51.530139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:51.530152 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:51.530409 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:51.530725 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:51.530728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:51.530730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:51.530891 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:51.531400 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:05:51.531407 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:05:51.531768 (+ 361us) client_negotiation.cc:769] Sending connection context
0924 07:05:51.531976 (+ 208us) client_negotiation.cc:240] Negotiation successful
0924 07:05:51.532279 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":204,"threads_started":1}
I20260924 07:05:51.534371 5262 heartbeater.cc:347] Connected to a master server at 127.3.217.62:36735
I20260924 07:05:51.534559 5262 heartbeater.cc:464] Registering TS with master...
I20260924 07:05:51.535166 5262 heartbeater.cc:511] Master 127.3.217.62:36735 requested a full tablet report, sending...
I20260924 07:05:51.536643 5059 ts_manager.cc:194] Registered new tserver with Master: 4bbe530d020145e1b47b5e84f0ce5f39 (127.3.217.1:35761)
I20260924 07:05:51.538092 5059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.1@KRBTEST.COM'} at 127.3.217.1:45531
WARNING: no policy specified for kudu/127.3.217.2@KRBTEST.COM; defaulting to no policy
I20260924 07:05:51.543242 5269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:51.536520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36735 (local address 127.3.217.1:59101)
0924 07:05:51.536757 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:51.536770 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:51.536850 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:51.537043 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:51.537046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:51.537216 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:05:51.537309 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:51.537314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.538192 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.538196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:51.538925 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:51.538933 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.539378 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.539382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:51.539594 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:51.540050 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:05:51.540064 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:05:51.540367 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:05:51.541839 (+ 1472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:51.541842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:51.541844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:51.542073 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:51.542320 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:51.542323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:51.542325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:51.542401 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:51.542745 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:05:51.542748 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:05:51.542829 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 07:05:51.542926 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 07:05:51.543049 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":73,"threads_started":1}
I20260924 07:05:51.543253 5108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:51.536769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:59101 (local address 127.3.217.62:36735)
0924 07:05:51.536869 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:51.536873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:51.536885 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:05:51.536915 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:51.536918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:51.536957 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:51.537019 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:51.537406 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.538075 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.539052 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.539282 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.539363 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:51.540486 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:51.540490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:51.540492 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:51.540514 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:51.541743 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:51.542168 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:51.542172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:51.542173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:51.542243 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:51.542476 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:51.542479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:51.542481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:51.542630 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:51.542708 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:51.543003 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 07:05:51.543099 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":45}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
I20260924 07:05:51.559579 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:36735
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:44097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:05:51.669154 5271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:05:51.669371 5271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:05:51.669427 5271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:05:51.673336 5271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:05:51.673388 5271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:05:51.673439 5271 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:05:51.673491 5271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:05:51.678578 5271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:44097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.217.62:36735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:05:51.679438 5271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:05:51.680318 5271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:05:51.689549 5277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:51.689571 5279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:51.689594 5276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:05:51.689752 5271 server_base.cc:1034] running on GCE node
I20260924 07:05:51.690234 5271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:05:51.690778 5271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:05:51.691942 5271 hybrid_clock.cc:648] HybridClock initialized: now 1790233551691928 us; error 28 us; skew 500 ppm
Sep 24 07:05:51 dist-test-slave-2p7j krb5kdc[5014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233551, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:05:51.694404 5271 init.cc:377] Logged in from keytab as kudu/127.3.217.2@KRBTEST.COM (short username kudu)
I20260924 07:05:51.695152 5271 webserver.cc:460] Webserver started at http://127.3.217.2:35711/ using document root <none> and password file <none>
I20260924 07:05:51.695678 5271 fs_manager.cc:359] Metadata directory not provided
I20260924 07:05:51.695739 5271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:05:51.695915 5271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:05:51.697701 5271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "834a3fe97ddf4cb5acb480d1384e8343"
format_stamp: "Formatted at 2026-09-24 07:05:51 on dist-test-slave-2p7j"
server_key: "f1db6c5774bc6bb4775eed7d0198fa22"
server_key_iv: "ec6a2b1c36d82b72f372bec158954fac"
server_key_version: "encryptionkey@0"
I20260924 07:05:51.698154 5271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "834a3fe97ddf4cb5acb480d1384e8343"
format_stamp: "Formatted at 2026-09-24 07:05:51 on dist-test-slave-2p7j"
server_key: "f1db6c5774bc6bb4775eed7d0198fa22"
server_key_iv: "ec6a2b1c36d82b72f372bec158954fac"
server_key_version: "encryptionkey@0"
I20260924 07:05:51.701431 5271 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:05:51.703841 5286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:51.705077 5271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:05:51.705181 5271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "834a3fe97ddf4cb5acb480d1384e8343"
format_stamp: "Formatted at 2026-09-24 07:05:51 on dist-test-slave-2p7j"
server_key: "f1db6c5774bc6bb4775eed7d0198fa22"
server_key_iv: "ec6a2b1c36d82b72f372bec158954fac"
server_key_version: "encryptionkey@0"
I20260924 07:05:51.705274 5271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:05:51.718086 5271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:05:51.720713 5271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:05:51.720860 5271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:05:51.722112 5271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:05:51.722160 5271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:51.722198 5271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:05:51.722211 5271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:05:51 dist-test-slave-2p7j krb5kdc[5014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233551, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:05:51.732322 5271 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:39199
I20260924 07:05:51.732374 5401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:39199 every 8 connection(s)
I20260924 07:05:51.733217 5271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:05:51.734896 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 5271
I20260924 07:05:51.735002 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:05:51.735294 3940 external_mini_cluster.cc:1428] Setting key dbf1467d5e96419e5d74c7572bb2d008
I20260924 07:05:51.736011 5108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:51.722778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:56813 (local address 127.3.217.62:36735)
0924 07:05:51.722952 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:51.722957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:51.723764 (+ 807us) server_negotiation.cc:407] Connection header received
0924 07:05:51.724681 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:51.724684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:51.724740 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:51.724795 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:51.726169 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.726928 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.728544 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.728764 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.728848 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:51.731394 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:51.731398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:51.731401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:51.731438 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:51.733306 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:51.733999 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:51.734003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:51.734004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:51.734096 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:51.734432 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:51.734434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:51.734436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:51.734605 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:51.734720 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:51.735655 (+ 935us) server_negotiation.cc:299] Negotiation successful
0924 07:05:51.735825 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 07:05:51.736353 5295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:51.723163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36735 (local address 127.3.217.2:56813)
0924 07:05:51.723622 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:51.723668 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:51.724454 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:51.725156 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:51.725165 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:51.725536 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:05:51.726043 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:51.726059 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.727028 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.727031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:51.728424 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:51.728434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.728909 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.728914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:51.729067 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:51.729545 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:05:51.729560 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:05:51.731217 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:05:51.733516 (+ 2299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:51.733523 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:51.733536 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:51.733866 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:51.734176 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:51.734179 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:51.734180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:51.734329 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:51.734740 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:05:51.734746 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:05:51.734989 (+ 243us) client_negotiation.cc:769] Sending connection context
0924 07:05:51.735178 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 07:05:51.735469 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":126,"threads_started":1}
I20260924 07:05:51.737401 5402 heartbeater.cc:347] Connected to a master server at 127.3.217.62:36735
I20260924 07:05:51.737532 5402 heartbeater.cc:464] Registering TS with master...
I20260924 07:05:51.737946 5402 heartbeater.cc:511] Master 127.3.217.62:36735 requested a full tablet report, sending...
I20260924 07:05:51.738906 5059 ts_manager.cc:194] Registered new tserver with Master: 834a3fe97ddf4cb5acb480d1384e8343 (127.3.217.2:39199)
I20260924 07:05:51.739596 5059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.2@KRBTEST.COM'} at 127.3.217.2:56813
WARNING: no policy specified for kudu/127.3.217.3@KRBTEST.COM; defaulting to no policy
I20260924 07:05:51.746693 5108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:51.739869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:53663 (local address 127.3.217.62:36735)
0924 07:05:51.739962 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:51.739966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:51.739978 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:05:51.740007 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:51.740009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:51.740043 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:51.740113 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:51.740565 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.741303 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.742072 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.742392 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.742498 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:51.743448 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:51.743453 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:51.743455 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:51.743480 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:51.745000 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:51.745420 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:51.745423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:51.745425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:51.745502 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:51.745874 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:51.745878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:51.745879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:51.746083 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:51.746188 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:51.746441 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 07:05:51.746543 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:05:51.746729 5409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:51.739352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36735 (local address 127.3.217.2:53663)
0924 07:05:51.739633 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:51.739647 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:51.739727 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:51.740130 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:51.740133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:51.740317 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:05:51.740456 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:51.740464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.741428 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.741431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:51.741969 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:51.741978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.742498 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.742501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:51.742586 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:51.742983 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:05:51.742997 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:05:51.743319 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:05:51.745114 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:51.745116 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:51.745118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:51.745315 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:51.745685 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:51.745688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:51.745691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:51.745785 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:51.746191 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:05:51.746195 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:05:51.746286 (+ 91us) client_negotiation.cc:769] Sending connection context
0924 07:05:51.746379 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 07:05:51.746516 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":5,"thread_start_us":94,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
I20260924 07:05:51.763121 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:36735
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:44097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:05:51.867856 5411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:05:51.868072 5411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:05:51.868111 5411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:05:51.872155 5411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:05:51.872215 5411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:05:51.872255 5411 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:05:51.872285 5411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:05:51.877593 5411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:44097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.217.62:36735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:05:51.878427 5411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:05:51.879308 5411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:05:51.888415 5417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:51.888404 5419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:51.888516 5416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:05:51.889129 5411 server_base.cc:1034] running on GCE node
I20260924 07:05:51.889724 5411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:05:51.890404 5411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:05:51.891583 5411 hybrid_clock.cc:648] HybridClock initialized: now 1790233551891558 us; error 36 us; skew 500 ppm
Sep 24 07:05:51 dist-test-slave-2p7j krb5kdc[5014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233551, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:05:51.894032 5411 init.cc:377] Logged in from keytab as kudu/127.3.217.3@KRBTEST.COM (short username kudu)
I20260924 07:05:51.894748 5411 webserver.cc:460] Webserver started at http://127.3.217.3:33553/ using document root <none> and password file <none>
I20260924 07:05:51.895233 5411 fs_manager.cc:359] Metadata directory not provided
I20260924 07:05:51.895295 5411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:05:51.895481 5411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:05:51.897447 5411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "626aa2ff5fba421592e83f48f4fec2ac"
format_stamp: "Formatted at 2026-09-24 07:05:51 on dist-test-slave-2p7j"
server_key: "4f7bfee6fdd0bb71dfe5cfc8ed4f184d"
server_key_iv: "52682a2810dd29ff0a58543f240a44b0"
server_key_version: "encryptionkey@0"
I20260924 07:05:51.897964 5411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "626aa2ff5fba421592e83f48f4fec2ac"
format_stamp: "Formatted at 2026-09-24 07:05:51 on dist-test-slave-2p7j"
server_key: "4f7bfee6fdd0bb71dfe5cfc8ed4f184d"
server_key_iv: "52682a2810dd29ff0a58543f240a44b0"
server_key_version: "encryptionkey@0"
I20260924 07:05:51.901335 5411 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:05:51.903714 5426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:51.905155 5411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:05:51.905282 5411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "626aa2ff5fba421592e83f48f4fec2ac"
format_stamp: "Formatted at 2026-09-24 07:05:51 on dist-test-slave-2p7j"
server_key: "4f7bfee6fdd0bb71dfe5cfc8ed4f184d"
server_key_iv: "52682a2810dd29ff0a58543f240a44b0"
server_key_version: "encryptionkey@0"
I20260924 07:05:51.905373 5411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:05:51.924679 5411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:05:51.927435 5411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:05:51.927620 5411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:05:51.928946 5411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:05:51.928992 5411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:51.929044 5411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:05:51.929073 5411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:51.938105 5411 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:36953
I20260924 07:05:51.938205 5541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:36953 every 8 connection(s)
I20260924 07:05:51.939009 5411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
Sep 24 07:05:51 dist-test-slave-2p7j krb5kdc[5014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233551, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:05:51.942623 5108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:51.929537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:58539 (local address 127.3.217.62:36735)
0924 07:05:51.929677 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:51.929681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:51.931574 (+ 1893us) server_negotiation.cc:407] Connection header received
0924 07:05:51.932974 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:51.932978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:51.933036 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:51.933131 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:51.934536 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.935378 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.936547 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.936885 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.937008 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:51.939721 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:51.939725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:51.939727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:51.939754 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:51.940813 (+ 1059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:51.941314 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:51.941316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:51.941318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:51.941404 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:51.941638 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:51.941640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:51.941642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:51.941806 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:51.941877 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:51.942381 (+ 504us) server_negotiation.cc:299] Negotiation successful
0924 07:05:51.942477 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:05:51.943323 5437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:51.930594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36735 (local address 127.3.217.3:58539)
0924 07:05:51.931318 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:51.931379 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:51.932714 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:51.933320 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:51.933329 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:51.933752 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:05:51.934398 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:51.934419 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.935478 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.935481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:51.936429 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:51.936439 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.937060 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.937065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:51.937270 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:51.937776 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:05:51.937796 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:05:51.939523 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:05:51.940990 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:51.940995 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:51.941005 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:51.941225 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:51.941468 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:51.941469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:51.941471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:51.941582 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:51.941923 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:05:51.941928 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:05:51.942128 (+ 200us) client_negotiation.cc:769] Sending connection context
0924 07:05:51.942290 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 07:05:51.942587 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":197,"threads_started":1}
I20260924 07:05:51.944396 5542 heartbeater.cc:347] Connected to a master server at 127.3.217.62:36735
I20260924 07:05:51.944520 5542 heartbeater.cc:464] Registering TS with master...
I20260924 07:05:51.944955 5542 heartbeater.cc:511] Master 127.3.217.62:36735 requested a full tablet report, sending...
I20260924 07:05:51.945869 5058 ts_manager.cc:194] Registered new tserver with Master: 626aa2ff5fba421592e83f48f4fec2ac (127.3.217.3:36953)
I20260924 07:05:51.946539 5058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.3@KRBTEST.COM'} at 127.3.217.3:58539
I20260924 07:05:51.948565 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 5411
I20260924 07:05:51.948647 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:05:51.948859 3940 external_mini_cluster.cc:1428] Setting key 6551d4ccd7fa915bf5cfe5e2c7653267
I20260924 07:05:51.950599 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:05:51.952890 5548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:51.946284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36735 (local address 127.3.217.3:55099)
0924 07:05:51.946556 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:51.946569 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:51.946659 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:51.946895 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:51.946897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:51.947084 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:05:51.947181 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:51.947186 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.948174 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.948177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:51.948781 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:51.948789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.949233 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.949236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:51.949321 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:51.949744 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:05:51.949757 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:05:51.950089 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:05:51.951558 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:51.951561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:51.951563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:51.951740 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:51.951970 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:51.951973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:51.951975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:51.952058 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:51.952407 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:05:51.952410 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:05:51.952492 (+ 82us) client_negotiation.cc:769] Sending connection context
0924 07:05:51.952592 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 07:05:51.952711 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":82,"threads_started":1}
I20260924 07:05:51.953203 5108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:51.946381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:55099 (local address 127.3.217.62:36735)
0924 07:05:51.946469 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:51.946472 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:51.946635 (+ 163us) server_negotiation.cc:407] Connection header received
0924 07:05:51.946755 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:51.946759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:51.946802 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:51.946872 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:51.947286 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.948046 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.948892 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.949131 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.949209 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:51.950189 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:51.950191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:51.950193 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:51.950212 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:51.951447 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:51.951821 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:51.951824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:51.951826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:51.951897 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:51.952160 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:51.952164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:51.952166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:51.952329 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:51.952403 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:51.952807 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 07:05:51.952998 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 07:05:51 dist-test-slave-2p7j krb5kdc[5014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233551, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 07:05:51 dist-test-slave-2p7j krb5kdc[5014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233551, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:05:51.974992 5108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:51.967064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59410 (local address 127.3.217.62:36735)
0924 07:05:51.967173 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:51.967176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:51.967190 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:05:51.967224 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:51.967226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:51.967260 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:51.967324 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:51.967972 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.968465 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.969266 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.969487 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.969570 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:51.971907 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:51.971910 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:51.971912 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:51.971932 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:51.973253 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:51.973622 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:51.973625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:51.973626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:51.973694 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:51.974030 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:51.974036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:51.974037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:51.974335 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:51.974494 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:51.974770 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 07:05:51.974873 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:05:51.978080 5058 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:05:51.981017 5058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:05:51.995267 5560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:51.989716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38394 (local address 127.3.217.1:35761)
0924 07:05:51.990164 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:51.990175 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:51.990675 (+ 500us) server_negotiation.cc:407] Connection header received
0924 07:05:51.990817 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:51.990822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:51.991016 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:51.991153 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:05:51.991463 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.992604 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.993881 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.994559 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.994666 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:51.994834 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:51.995011 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 07:05:51.995116 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":121,"threads_started":1}
I20260924 07:05:51.995431 5559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:51.990287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:39199 (local address 127.0.0.1:43936)
0924 07:05:51.990880 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:51.990892 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:51.990954 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:51.991765 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:51.991768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:51.991785 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:05:51.991881 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:51.991888 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.993031 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.993034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:51.994179 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:51.994186 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.994964 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.994969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:51.995071 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:51.995094 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:05:51.995170 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 07:05:51.995294 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":3,"thread_start_us":78,"threads_started":1}
I20260924 07:05:51.995678 5561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:51.989561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:35761 (local address 127.0.0.1:38394)
0924 07:05:51.990497 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:51.990541 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:51.990679 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:51.991137 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:51.991139 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:51.991157 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:05:51.991306 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:51.991313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.992878 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.992881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:51.993761 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:51.993771 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.994677 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.994681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:51.994808 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:51.994850 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 07:05:51.995461 (+ 611us) client_negotiation.cc:240] Negotiation successful
0924 07:05:51.995552 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":25,"thread_start_us":76,"threads_started":1}
I20260924 07:05:51.995697 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:51.990326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54912 (local address 127.3.217.3:36953)
0924 07:05:51.990703 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:51.990714 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:51.990734 (+ 20us) server_negotiation.cc:407] Connection header received
0924 07:05:51.990816 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:51.990821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:51.990975 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:51.991099 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:05:51.991384 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.992558 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.994188 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.994913 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.995281 (+ 368us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:51.995418 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:51.995467 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 07:05:51.995570 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":112,"threads_started":1}
I20260924 07:05:51.996084 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:51.990437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43936 (local address 127.3.217.2:39199)
0924 07:05:51.991390 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:51.991402 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:51.991422 (+ 20us) server_negotiation.cc:407] Connection header received
0924 07:05:51.991485 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:51.991488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:51.991654 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:51.991781 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:05:51.991974 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.992910 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.994267 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.994859 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.994963 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:51.995107 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:51.995194 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 07:05:51.995280 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":860,"thread_start_us":81,"threads_started":1}
I20260924 07:05:51.996291 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:51.989934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:36953 (local address 127.0.0.1:54912)
0924 07:05:51.990571 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:51.990583 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:51.990680 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:51.991109 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:51.991114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:51.991135 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:05:51.991283 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:51.991289 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.992675 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.992678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:51.994070 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:51.994080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:51.995027 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:51.995031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:51.995103 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:51.995121 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 07:05:51.995159 (+ 38us) client_negotiation.cc:240] Negotiation successful
0924 07:05:51.996175 (+ 1016us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":952,"thread_start_us":84,"threads_started":1}
I20260924 07:05:51.998404 5336 tablet_service.cc:1467] Processing CreateTablet for tablet 253968be5f084ede8677eadc77c66dc8 (DEFAULT_TABLE table=test-table [id=c1023327809746b88441667a37f1c523]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:05:51.998564 5476 tablet_service.cc:1467] Processing CreateTablet for tablet 253968be5f084ede8677eadc77c66dc8 (DEFAULT_TABLE table=test-table [id=c1023327809746b88441667a37f1c523]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:05:51.998548 5196 tablet_service.cc:1467] Processing CreateTablet for tablet 253968be5f084ede8677eadc77c66dc8 (DEFAULT_TABLE table=test-table [id=c1023327809746b88441667a37f1c523]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:05:51.999331 5336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253968be5f084ede8677eadc77c66dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:51.999567 5476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253968be5f084ede8677eadc77c66dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:51.999559 5196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253968be5f084ede8677eadc77c66dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:52.005859 5565 tablet_bootstrap.cc:492] T 253968be5f084ede8677eadc77c66dc8 P 834a3fe97ddf4cb5acb480d1384e8343: Bootstrap starting.
I20260924 07:05:52.008114 5565 tablet_bootstrap.cc:654] T 253968be5f084ede8677eadc77c66dc8 P 834a3fe97ddf4cb5acb480d1384e8343: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:52.008291 5567 tablet_bootstrap.cc:492] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39: Bootstrap starting.
I20260924 07:05:52.008904 5565 log.cc:824] T 253968be5f084ede8677eadc77c66dc8 P 834a3fe97ddf4cb5acb480d1384e8343: Log is configured to *not* fsync() on all Append() calls
I20260924 07:05:52.010283 5567 tablet_bootstrap.cc:654] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:52.010761 5565 tablet_bootstrap.cc:492] T 253968be5f084ede8677eadc77c66dc8 P 834a3fe97ddf4cb5acb480d1384e8343: No bootstrap required, opened a new log
I20260924 07:05:52.010932 5565 ts_tablet_manager.cc:1397] T 253968be5f084ede8677eadc77c66dc8 P 834a3fe97ddf4cb5acb480d1384e8343: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:05:52.010957 5567 log.cc:824] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39: Log is configured to *not* fsync() on all Append() calls
I20260924 07:05:52.011587 5566 tablet_bootstrap.cc:492] T 253968be5f084ede8677eadc77c66dc8 P 626aa2ff5fba421592e83f48f4fec2ac: Bootstrap starting.
I20260924 07:05:52.013101 5566 tablet_bootstrap.cc:654] T 253968be5f084ede8677eadc77c66dc8 P 626aa2ff5fba421592e83f48f4fec2ac: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:52.013723 5565 raft_consensus.cc:348] T 253968be5f084ede8677eadc77c66dc8 P 834a3fe97ddf4cb5acb480d1384e8343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.013921 5565 raft_consensus.cc:374] T 253968be5f084ede8677eadc77c66dc8 P 834a3fe97ddf4cb5acb480d1384e8343 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:52.013953 5565 raft_consensus.cc:729] T 253968be5f084ede8677eadc77c66dc8 P 834a3fe97ddf4cb5acb480d1384e8343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 834a3fe97ddf4cb5acb480d1384e8343, State: Initialized, Role: FOLLOWER
I20260924 07:05:52.014070 5566 log.cc:824] T 253968be5f084ede8677eadc77c66dc8 P 626aa2ff5fba421592e83f48f4fec2ac: Log is configured to *not* fsync() on all Append() calls
I20260924 07:05:52.014357 5565 consensus_queue.cc:260] T 253968be5f084ede8677eadc77c66dc8 P 834a3fe97ddf4cb5acb480d1384e8343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.014985 5402 heartbeater.cc:503] Master 127.3.217.62:36735 was elected leader, sending a full tablet report...
I20260924 07:05:52.015218 5565 ts_tablet_manager.cc:1428] T 253968be5f084ede8677eadc77c66dc8 P 834a3fe97ddf4cb5acb480d1384e8343: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260924 07:05:52.015704 5567 tablet_bootstrap.cc:492] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39: No bootstrap required, opened a new log
I20260924 07:05:52.015894 5567 ts_tablet_manager.cc:1397] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 07:05:52.019140 5567 raft_consensus.cc:348] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.019362 5567 raft_consensus.cc:374] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:52.019425 5567 raft_consensus.cc:729] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bbe530d020145e1b47b5e84f0ce5f39, State: Initialized, Role: FOLLOWER
I20260924 07:05:52.019819 5567 consensus_queue.cc:260] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.020452 5262 heartbeater.cc:503] Master 127.3.217.62:36735 was elected leader, sending a full tablet report...
I20260924 07:05:52.020681 5567 ts_tablet_manager.cc:1428] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 07:05:52.021878 5566 tablet_bootstrap.cc:492] T 253968be5f084ede8677eadc77c66dc8 P 626aa2ff5fba421592e83f48f4fec2ac: No bootstrap required, opened a new log
I20260924 07:05:52.022053 5566 ts_tablet_manager.cc:1397] T 253968be5f084ede8677eadc77c66dc8 P 626aa2ff5fba421592e83f48f4fec2ac: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20260924 07:05:52.024927 5566 raft_consensus.cc:348] T 253968be5f084ede8677eadc77c66dc8 P 626aa2ff5fba421592e83f48f4fec2ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.025118 5566 raft_consensus.cc:374] T 253968be5f084ede8677eadc77c66dc8 P 626aa2ff5fba421592e83f48f4fec2ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:52.025168 5566 raft_consensus.cc:729] T 253968be5f084ede8677eadc77c66dc8 P 626aa2ff5fba421592e83f48f4fec2ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626aa2ff5fba421592e83f48f4fec2ac, State: Initialized, Role: FOLLOWER
I20260924 07:05:52.025600 5566 consensus_queue.cc:260] T 253968be5f084ede8677eadc77c66dc8 P 626aa2ff5fba421592e83f48f4fec2ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.026214 5542 heartbeater.cc:503] Master 127.3.217.62:36735 was elected leader, sending a full tablet report...
I20260924 07:05:52.026427 5566 ts_tablet_manager.cc:1428] T 253968be5f084ede8677eadc77c66dc8 P 626aa2ff5fba421592e83f48f4fec2ac: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260924 07:05:52.031318 5263 tablet.cc:2369] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:05:52.189574 5572 raft_consensus.cc:482] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:05:52.189875 5572 raft_consensus.cc:504] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
W20260924 07:05:52.189941 5543 tablet.cc:2369] T 253968be5f084ede8677eadc77c66dc8 P 626aa2ff5fba421592e83f48f4fec2ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:05:52.191330 5572 leader_election.cc:288] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 834a3fe97ddf4cb5acb480d1384e8343 (127.3.217.2:39199), 626aa2ff5fba421592e83f48f4fec2ac (127.3.217.3:36953)
I20260924 07:05:52.195258 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:52.191709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:55287 (local address 127.3.217.2:39199)
0924 07:05:52.191830 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:52.191834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:52.192047 (+ 213us) server_negotiation.cc:407] Connection header received
0924 07:05:52.192220 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:52.192223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:52.192280 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:52.192346 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:05:52.192627 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.193119 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.194313 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.194684 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.194785 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:52.194844 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:52.195070 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 07:05:52.195137 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:05:52.195269 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:52.191917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:34171 (local address 127.3.217.3:36953)
0924 07:05:52.192049 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:52.192054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:52.192071 (+ 17us) server_negotiation.cc:407] Connection header received
0924 07:05:52.192226 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:52.192229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:52.192272 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:52.192340 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:05:52.192612 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.193126 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.194284 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.194687 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.194790 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:52.194852 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:52.195079 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 07:05:52.195150 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 07:05:52.195290 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:52.191699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:36953 (local address 127.3.217.1:34171)
0924 07:05:52.191945 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:52.191959 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:52.192117 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:52.192367 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:52.192370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:52.192385 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:05:52.192512 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:52.192517 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.193299 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.193302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:52.194159 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:52.194168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.194807 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.194811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:52.194963 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:52.194981 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 07:05:52.195045 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 07:05:52.195119 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":114,"spinlock_wait_cycles":18688,"thread_start_us":61,"threads_started":1}
I20260924 07:05:52.195295 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:52.191619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:39199 (local address 127.3.217.1:55287)
0924 07:05:52.191943 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:52.191960 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:52.192093 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:52.192376 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:52.192379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:52.192396 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:05:52.192521 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:52.192526 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.193231 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.193235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:52.194165 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:52.194176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.194803 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.194808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:52.194934 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:52.194970 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 07:05:52.195042 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 07:05:52.195126 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":61,"thread_start_us":73,"threads_started":1}
I20260924 07:05:52.196061 5356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253968be5f084ede8677eadc77c66dc8" candidate_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "834a3fe97ddf4cb5acb480d1384e8343" is_pre_election: true
I20260924 07:05:52.196106 5496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253968be5f084ede8677eadc77c66dc8" candidate_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626aa2ff5fba421592e83f48f4fec2ac" is_pre_election: true
I20260924 07:05:52.196380 5496 raft_consensus.cc:2454] T 253968be5f084ede8677eadc77c66dc8 P 626aa2ff5fba421592e83f48f4fec2ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bbe530d020145e1b47b5e84f0ce5f39 in term 0.
I20260924 07:05:52.196415 5356 raft_consensus.cc:2454] T 253968be5f084ede8677eadc77c66dc8 P 834a3fe97ddf4cb5acb480d1384e8343 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bbe530d020145e1b47b5e84f0ce5f39 in term 0.
I20260924 07:05:52.196853 5148 leader_election.cc:302] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bbe530d020145e1b47b5e84f0ce5f39, 626aa2ff5fba421592e83f48f4fec2ac; no voters:
I20260924 07:05:52.197132 5572 raft_consensus.cc:2781] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:05:52.197224 5572 raft_consensus.cc:482] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:05:52.197271 5572 raft_consensus.cc:3037] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:52.198298 5572 raft_consensus.cc:504] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.198652 5572 leader_election.cc:288] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39 [CANDIDATE]: Term 1 election: Requested vote from peers 834a3fe97ddf4cb5acb480d1384e8343 (127.3.217.2:39199), 626aa2ff5fba421592e83f48f4fec2ac (127.3.217.3:36953)
I20260924 07:05:52.198982 5356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253968be5f084ede8677eadc77c66dc8" candidate_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "834a3fe97ddf4cb5acb480d1384e8343"
I20260924 07:05:52.198995 5496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253968be5f084ede8677eadc77c66dc8" candidate_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626aa2ff5fba421592e83f48f4fec2ac"
I20260924 07:05:52.199087 5356 raft_consensus.cc:3037] T 253968be5f084ede8677eadc77c66dc8 P 834a3fe97ddf4cb5acb480d1384e8343 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:52.199121 5496 raft_consensus.cc:3037] T 253968be5f084ede8677eadc77c66dc8 P 626aa2ff5fba421592e83f48f4fec2ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:52.200345 5496 raft_consensus.cc:2454] T 253968be5f084ede8677eadc77c66dc8 P 626aa2ff5fba421592e83f48f4fec2ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bbe530d020145e1b47b5e84f0ce5f39 in term 1.
I20260924 07:05:52.200628 5148 leader_election.cc:302] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bbe530d020145e1b47b5e84f0ce5f39, 626aa2ff5fba421592e83f48f4fec2ac; no voters:
I20260924 07:05:52.200830 5572 raft_consensus.cc:2781] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:05:52.201002 5572 raft_consensus.cc:686] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 1 LEADER]: Becoming Leader. State: Replica: 4bbe530d020145e1b47b5e84f0ce5f39, State: Running, Role: LEADER
I20260924 07:05:52.201040 5356 raft_consensus.cc:2454] T 253968be5f084ede8677eadc77c66dc8 P 834a3fe97ddf4cb5acb480d1384e8343 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bbe530d020145e1b47b5e84f0ce5f39 in term 1.
I20260924 07:05:52.201330 5572 consensus_queue.cc:237] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.204447 5058 catalog_manager.cc:5495] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bbe530d020145e1b47b5e84f0ce5f39 (127.3.217.1). New cstate: current_term: 1 leader_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } health_report { overall_health: UNKNOWN } } }
I20260924 07:05:52.232085 5560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:52.228692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38402 (local address 127.3.217.1:35761)
0924 07:05:52.228808 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:52.228813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:52.228828 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:05:52.229037 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:52.229040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:52.229093 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:52.229168 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:05:52.229409 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.229897 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.230747 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.230963 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.231047 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:52.231343 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:05:52.231719 (+ 376us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:05:52.231815 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:52.231905 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 07:05:52.231960 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 07:05:52.234505 5403 tablet.cc:2369] T 253968be5f084ede8677eadc77c66dc8 P 834a3fe97ddf4cb5acb480d1384e8343: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:05:52.237824 5356 raft_consensus.cc:1261] T 253968be5f084ede8677eadc77c66dc8 P 834a3fe97ddf4cb5acb480d1384e8343 [term 1 FOLLOWER]: Refusing update from remote peer 4bbe530d020145e1b47b5e84f0ce5f39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:05:52.237824 5496 raft_consensus.cc:1261] T 253968be5f084ede8677eadc77c66dc8 P 626aa2ff5fba421592e83f48f4fec2ac [term 1 FOLLOWER]: Refusing update from remote peer 4bbe530d020145e1b47b5e84f0ce5f39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:05:52.238564 5572 consensus_queue.cc:1035] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:05:52.238793 5576 consensus_queue.cc:1035] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:05:52.246733 5580 mvcc.cc:205] Tried to move back new op lower bound from 7332796629959290880 to 7332796629819346944. Current Snapshot: MvccSnapshot[applied={T|T < 7332796629959290880}]
I20260924 07:05:52.247684 5581 mvcc.cc:205] Tried to move back new op lower bound from 7332796629959290880 to 7332796629819346944. Current Snapshot: MvccSnapshot[applied={T|T < 7332796629959290880 or (T in {7332796629959290880})}]
I20260924 07:05:52.257228 5058 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:05:52.257478 5058 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:05:52.260408 5058 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753: Sending DeleteTablet for 3 replicas of tablet 253968be5f084ede8677eadc77c66dc8
I20260924 07:05:52.261278 5196 tablet_service.cc:1514] Processing DeleteTablet for tablet 253968be5f084ede8677eadc77c66dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:05:52 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:38394
I20260924 07:05:52.261600 5476 tablet_service.cc:1514] Processing DeleteTablet for tablet 253968be5f084ede8677eadc77c66dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:05:52 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:54912
I20260924 07:05:52.261821 5589 tablet_replica.cc:330] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39: stopping tablet replica
I20260924 07:05:52.261862 5336 tablet_service.cc:1514] Processing DeleteTablet for tablet 253968be5f084ede8677eadc77c66dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:05:52 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:43936
I20260924 07:05:52.262202 5589 raft_consensus.cc:2229] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:05:52.262290 5590 tablet_replica.cc:330] T 253968be5f084ede8677eadc77c66dc8 P 626aa2ff5fba421592e83f48f4fec2ac: stopping tablet replica
I20260924 07:05:52.262535 5590 raft_consensus.cc:2229] T 253968be5f084ede8677eadc77c66dc8 P 626aa2ff5fba421592e83f48f4fec2ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:05:52.262391 5058 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 07:05:52.262820 5590 raft_consensus.cc:2258] T 253968be5f084ede8677eadc77c66dc8 P 626aa2ff5fba421592e83f48f4fec2ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:05:52.262926 5589 raft_consensus.cc:2258] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 07:05:52.263116 5058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:05:52.264318 5590 ts_tablet_manager.cc:1905] T 253968be5f084ede8677eadc77c66dc8 P 626aa2ff5fba421592e83f48f4fec2ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:05:52.264318 5589 ts_tablet_manager.cc:1905] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:05:52.267241 5589 ts_tablet_manager.cc:1918] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:05:52.267383 5589 log.cc:1196] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-0/wal/wals/253968be5f084ede8677eadc77c66dc8
I20260924 07:05:52.267748 5589 ts_tablet_manager.cc:1939] T 253968be5f084ede8677eadc77c66dc8 P 4bbe530d020145e1b47b5e84f0ce5f39: Deleting consensus metadata
I20260924 07:05:52.268257 5476 tablet_service.cc:1467] Processing CreateTablet for tablet bb7f9b2bfc4449b5ab44ebf970bdead0 (DEFAULT_TABLE table=test-table [id=bd4c1857d93e428eab02848714715d9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:05:52.268496 5476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb7f9b2bfc4449b5ab44ebf970bdead0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:52.268523 5336 tablet_service.cc:1467] Processing CreateTablet for tablet bb7f9b2bfc4449b5ab44ebf970bdead0 (DEFAULT_TABLE table=test-table [id=bd4c1857d93e428eab02848714715d9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:05:52.268743 5336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb7f9b2bfc4449b5ab44ebf970bdead0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:52.268980 5591 tablet_replica.cc:330] T 253968be5f084ede8677eadc77c66dc8 P 834a3fe97ddf4cb5acb480d1384e8343: stopping tablet replica
I20260924 07:05:52.269232 5591 raft_consensus.cc:2229] T 253968be5f084ede8677eadc77c66dc8 P 834a3fe97ddf4cb5acb480d1384e8343 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:05:52.269479 5591 raft_consensus.cc:2258] T 253968be5f084ede8677eadc77c66dc8 P 834a3fe97ddf4cb5acb480d1384e8343 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:05:52.270675 5566 tablet_bootstrap.cc:492] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 626aa2ff5fba421592e83f48f4fec2ac: Bootstrap starting.
I20260924 07:05:52.268270 5196 tablet_service.cc:1467] Processing CreateTablet for tablet bb7f9b2bfc4449b5ab44ebf970bdead0 (DEFAULT_TABLE table=test-table [id=bd4c1857d93e428eab02848714715d9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:05:52.271304 5196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb7f9b2bfc4449b5ab44ebf970bdead0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:52.271554 5566 tablet_bootstrap.cc:654] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 626aa2ff5fba421592e83f48f4fec2ac: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:52.271775 5565 tablet_bootstrap.cc:492] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 834a3fe97ddf4cb5acb480d1384e8343: Bootstrap starting.
I20260924 07:05:52.272567 5566 tablet_bootstrap.cc:492] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 626aa2ff5fba421592e83f48f4fec2ac: No bootstrap required, opened a new log
I20260924 07:05:52.272634 5566 ts_tablet_manager.cc:1397] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 626aa2ff5fba421592e83f48f4fec2ac: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 07:05:52.273218 5565 tablet_bootstrap.cc:654] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 834a3fe97ddf4cb5acb480d1384e8343: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:52.273160 5566 raft_consensus.cc:348] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 626aa2ff5fba421592e83f48f4fec2ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.273347 5566 raft_consensus.cc:374] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 626aa2ff5fba421592e83f48f4fec2ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:52.273381 5566 raft_consensus.cc:729] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 626aa2ff5fba421592e83f48f4fec2ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626aa2ff5fba421592e83f48f4fec2ac, State: Initialized, Role: FOLLOWER
I20260924 07:05:52.273536 5566 consensus_queue.cc:260] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 626aa2ff5fba421592e83f48f4fec2ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.273815 5566 ts_tablet_manager.cc:1428] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 626aa2ff5fba421592e83f48f4fec2ac: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 07:05:52.273969 5567 tablet_bootstrap.cc:492] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39: Bootstrap starting.
I20260924 07:05:52.275388 5567 tablet_bootstrap.cc:654] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:52.276238 5043 catalog_manager.cc:4841] TS 4bbe530d020145e1b47b5e84f0ce5f39 (127.3.217.1:35761): tablet 253968be5f084ede8677eadc77c66dc8 (table test-table [id=c1023327809746b88441667a37f1c523]) successfully deleted
I20260924 07:05:52.276976 5567 tablet_bootstrap.cc:492] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39: No bootstrap required, opened a new log
I20260924 07:05:52.277077 5567 ts_tablet_manager.cc:1397] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:05:52.277580 5567 raft_consensus.cc:348] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.277719 5567 raft_consensus.cc:374] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:52.277756 5567 raft_consensus.cc:729] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bbe530d020145e1b47b5e84f0ce5f39, State: Initialized, Role: FOLLOWER
I20260924 07:05:52.277910 5567 consensus_queue.cc:260] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.278209 5567 ts_tablet_manager.cc:1428] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:05:52.280896 5590 ts_tablet_manager.cc:1918] T 253968be5f084ede8677eadc77c66dc8 P 626aa2ff5fba421592e83f48f4fec2ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:05:52.281085 5590 log.cc:1196] T 253968be5f084ede8677eadc77c66dc8 P 626aa2ff5fba421592e83f48f4fec2ac: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/253968be5f084ede8677eadc77c66dc8
I20260924 07:05:52.281423 5590 ts_tablet_manager.cc:1939] T 253968be5f084ede8677eadc77c66dc8 P 626aa2ff5fba421592e83f48f4fec2ac: Deleting consensus metadata
I20260924 07:05:52.282430 5591 ts_tablet_manager.cc:1905] T 253968be5f084ede8677eadc77c66dc8 P 834a3fe97ddf4cb5acb480d1384e8343: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:05:52.284564 5591 ts_tablet_manager.cc:1918] T 253968be5f084ede8677eadc77c66dc8 P 834a3fe97ddf4cb5acb480d1384e8343: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:05:52.284715 5591 log.cc:1196] T 253968be5f084ede8677eadc77c66dc8 P 834a3fe97ddf4cb5acb480d1384e8343: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/253968be5f084ede8677eadc77c66dc8
I20260924 07:05:52.285454 5044 catalog_manager.cc:4841] TS 626aa2ff5fba421592e83f48f4fec2ac (127.3.217.3:36953): tablet 253968be5f084ede8677eadc77c66dc8 (table test-table [id=c1023327809746b88441667a37f1c523]) successfully deleted
I20260924 07:05:52.285750 5591 ts_tablet_manager.cc:1939] T 253968be5f084ede8677eadc77c66dc8 P 834a3fe97ddf4cb5acb480d1384e8343: Deleting consensus metadata
I20260924 07:05:52.285944 5565 tablet_bootstrap.cc:492] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 834a3fe97ddf4cb5acb480d1384e8343: No bootstrap required, opened a new log
I20260924 07:05:52.286017 5565 ts_tablet_manager.cc:1397] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 834a3fe97ddf4cb5acb480d1384e8343: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260924 07:05:52.286427 5565 raft_consensus.cc:348] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 834a3fe97ddf4cb5acb480d1384e8343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.286536 5565 raft_consensus.cc:374] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 834a3fe97ddf4cb5acb480d1384e8343 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:52.286574 5565 raft_consensus.cc:729] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 834a3fe97ddf4cb5acb480d1384e8343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 834a3fe97ddf4cb5acb480d1384e8343, State: Initialized, Role: FOLLOWER
I20260924 07:05:52.286685 5565 consensus_queue.cc:260] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 834a3fe97ddf4cb5acb480d1384e8343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.286927 5565 ts_tablet_manager.cc:1428] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 834a3fe97ddf4cb5acb480d1384e8343: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:05:52.287442 5043 catalog_manager.cc:4841] TS 834a3fe97ddf4cb5acb480d1384e8343 (127.3.217.2:39199): tablet 253968be5f084ede8677eadc77c66dc8 (table test-table [id=c1023327809746b88441667a37f1c523]) successfully deleted
I20260924 07:05:52.345371 5585 raft_consensus.cc:482] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:05:52.345521 5585 raft_consensus.cc:504] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.345944 5585 leader_election.cc:288] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 834a3fe97ddf4cb5acb480d1384e8343 (127.3.217.2:39199), 626aa2ff5fba421592e83f48f4fec2ac (127.3.217.3:36953)
I20260924 07:05:52.346323 5496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb7f9b2bfc4449b5ab44ebf970bdead0" candidate_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626aa2ff5fba421592e83f48f4fec2ac" is_pre_election: true
I20260924 07:05:52.346323 5356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb7f9b2bfc4449b5ab44ebf970bdead0" candidate_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "834a3fe97ddf4cb5acb480d1384e8343" is_pre_election: true
I20260924 07:05:52.346474 5496 raft_consensus.cc:2454] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 626aa2ff5fba421592e83f48f4fec2ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bbe530d020145e1b47b5e84f0ce5f39 in term 0.
I20260924 07:05:52.346486 5356 raft_consensus.cc:2454] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 834a3fe97ddf4cb5acb480d1384e8343 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bbe530d020145e1b47b5e84f0ce5f39 in term 0.
I20260924 07:05:52.346728 5148 leader_election.cc:302] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bbe530d020145e1b47b5e84f0ce5f39, 626aa2ff5fba421592e83f48f4fec2ac; no voters:
I20260924 07:05:52.346897 5585 raft_consensus.cc:2781] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:05:52.346961 5585 raft_consensus.cc:482] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:05:52.347003 5585 raft_consensus.cc:3037] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:52.348196 5585 raft_consensus.cc:504] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.348553 5585 leader_election.cc:288] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39 [CANDIDATE]: Term 1 election: Requested vote from peers 834a3fe97ddf4cb5acb480d1384e8343 (127.3.217.2:39199), 626aa2ff5fba421592e83f48f4fec2ac (127.3.217.3:36953)
I20260924 07:05:52.348879 5356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb7f9b2bfc4449b5ab44ebf970bdead0" candidate_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "834a3fe97ddf4cb5acb480d1384e8343"
I20260924 07:05:52.348876 5496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb7f9b2bfc4449b5ab44ebf970bdead0" candidate_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626aa2ff5fba421592e83f48f4fec2ac"
I20260924 07:05:52.348991 5356 raft_consensus.cc:3037] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 834a3fe97ddf4cb5acb480d1384e8343 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:52.349000 5496 raft_consensus.cc:3037] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 626aa2ff5fba421592e83f48f4fec2ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:52.350035 5356 raft_consensus.cc:2454] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 834a3fe97ddf4cb5acb480d1384e8343 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bbe530d020145e1b47b5e84f0ce5f39 in term 1.
I20260924 07:05:52.350337 5147 leader_election.cc:302] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bbe530d020145e1b47b5e84f0ce5f39, 834a3fe97ddf4cb5acb480d1384e8343; no voters:
I20260924 07:05:52.350495 5585 raft_consensus.cc:2781] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:05:52.350487 5496 raft_consensus.cc:2454] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 626aa2ff5fba421592e83f48f4fec2ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bbe530d020145e1b47b5e84f0ce5f39 in term 1.
I20260924 07:05:52.350569 5585 raft_consensus.cc:686] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 1 LEADER]: Becoming Leader. State: Replica: 4bbe530d020145e1b47b5e84f0ce5f39, State: Running, Role: LEADER
I20260924 07:05:52.350690 5585 consensus_queue.cc:237] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.352360 5058 catalog_manager.cc:5495] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bbe530d020145e1b47b5e84f0ce5f39 (127.3.217.1). New cstate: current_term: 1 leader_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } health_report { overall_health: UNKNOWN } } }
Sep 24 07:05:52 dist-test-slave-2p7j krb5kdc[5014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233551, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:05:52.399338 5108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:52.389167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59426 (local address 127.3.217.62:36735)
0924 07:05:52.389270 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:52.389274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:52.389440 (+ 166us) server_negotiation.cc:407] Connection header received
0924 07:05:52.389568 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:52.389570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:52.389605 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:52.389686 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:52.390440 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.390951 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.392194 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.392541 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.392638 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:52.395719 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:52.395723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:52.395726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:52.395755 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:52.397588 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:52.398134 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:52.398137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:52.398138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:52.398219 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:52.398480 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:52.398482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:52.398483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:52.398638 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:52.398715 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:52.399044 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 07:05:52.399186 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":29}
I20260924 07:05:52.399362 5601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:52.389072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36735 (local address 127.0.0.1:59426)
0924 07:05:52.389353 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:52.389369 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:52.389484 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:52.389720 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:52.389723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:52.390209 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:05:52.390323 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:52.390330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.391053 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.391057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:52.391976 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:52.391987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.392668 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.392671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:52.392763 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:52.393537 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:05:52.393648 (+ 111us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:05:52.395444 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:05:52.397702 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:52.397711 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:52.397717 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:52.398019 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:52.398305 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:52.398308 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:52.398310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:52.398404 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:52.398719 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:05:52.398726 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:05:52.398847 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 07:05:52.398968 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 07:05:52.399127 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":93,"threads_started":1}
I20260924 07:05:52.403684 5058 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:59426:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 07:05:52.404510 5058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:05:52.409832 5196 tablet_service.cc:1467] Processing CreateTablet for tablet e114c23269424323bd004532a0c2d2d0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b36a5cb9d25e415a9465b6ead69bbea6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:05:52.409940 5476 tablet_service.cc:1467] Processing CreateTablet for tablet e114c23269424323bd004532a0c2d2d0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b36a5cb9d25e415a9465b6ead69bbea6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:05:52.410208 5196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e114c23269424323bd004532a0c2d2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:52.410218 5476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e114c23269424323bd004532a0c2d2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:52.412886 5567 tablet_bootstrap.cc:492] T e114c23269424323bd004532a0c2d2d0 P 4bbe530d020145e1b47b5e84f0ce5f39: Bootstrap starting.
I20260924 07:05:52.412899 5566 tablet_bootstrap.cc:492] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac: Bootstrap starting.
I20260924 07:05:52.413208 5336 tablet_service.cc:1467] Processing CreateTablet for tablet e114c23269424323bd004532a0c2d2d0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b36a5cb9d25e415a9465b6ead69bbea6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:05:52.413554 5336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e114c23269424323bd004532a0c2d2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:52.413720 5567 tablet_bootstrap.cc:654] T e114c23269424323bd004532a0c2d2d0 P 4bbe530d020145e1b47b5e84f0ce5f39: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:52.413790 5566 tablet_bootstrap.cc:654] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:52.415117 5566 tablet_bootstrap.cc:492] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac: No bootstrap required, opened a new log
I20260924 07:05:52.415198 5566 ts_tablet_manager.cc:1397] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 07:05:52.415630 5566 raft_consensus.cc:348] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.415746 5566 raft_consensus.cc:374] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:52.415786 5566 raft_consensus.cc:729] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626aa2ff5fba421592e83f48f4fec2ac, State: Initialized, Role: FOLLOWER
I20260924 07:05:52.415903 5566 consensus_queue.cc:260] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.416150 5566 ts_tablet_manager.cc:1428] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:05:52.416363 5565 tablet_bootstrap.cc:492] T e114c23269424323bd004532a0c2d2d0 P 834a3fe97ddf4cb5acb480d1384e8343: Bootstrap starting.
I20260924 07:05:52.416361 5573 tablet_replica.cc:439] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } } }
I20260924 07:05:52.416489 5573 tablet_replica.cc:442] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:05:52.417088 5567 tablet_bootstrap.cc:492] T e114c23269424323bd004532a0c2d2d0 P 4bbe530d020145e1b47b5e84f0ce5f39: No bootstrap required, opened a new log
I20260924 07:05:52.417177 5567 ts_tablet_manager.cc:1397] T e114c23269424323bd004532a0c2d2d0 P 4bbe530d020145e1b47b5e84f0ce5f39: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 07:05:52.417309 5565 tablet_bootstrap.cc:654] T e114c23269424323bd004532a0c2d2d0 P 834a3fe97ddf4cb5acb480d1384e8343: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:52.417520 5567 raft_consensus.cc:348] T e114c23269424323bd004532a0c2d2d0 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.417620 5567 raft_consensus.cc:374] T e114c23269424323bd004532a0c2d2d0 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:52.417647 5567 raft_consensus.cc:729] T e114c23269424323bd004532a0c2d2d0 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bbe530d020145e1b47b5e84f0ce5f39, State: Initialized, Role: FOLLOWER
I20260924 07:05:52.417750 5567 consensus_queue.cc:260] T e114c23269424323bd004532a0c2d2d0 P 4bbe530d020145e1b47b5e84f0ce5f39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.418002 5567 ts_tablet_manager.cc:1428] T e114c23269424323bd004532a0c2d2d0 P 4bbe530d020145e1b47b5e84f0ce5f39: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:05:52.418177 5576 tablet_replica.cc:439] T e114c23269424323bd004532a0c2d2d0 P 4bbe530d020145e1b47b5e84f0ce5f39: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } } }
I20260924 07:05:52.418280 5576 tablet_replica.cc:442] T e114c23269424323bd004532a0c2d2d0 P 4bbe530d020145e1b47b5e84f0ce5f39: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:05:52.420058 5565 tablet_bootstrap.cc:492] T e114c23269424323bd004532a0c2d2d0 P 834a3fe97ddf4cb5acb480d1384e8343: No bootstrap required, opened a new log
I20260924 07:05:52.420158 5565 ts_tablet_manager.cc:1397] T e114c23269424323bd004532a0c2d2d0 P 834a3fe97ddf4cb5acb480d1384e8343: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 07:05:52.420550 5565 raft_consensus.cc:348] T e114c23269424323bd004532a0c2d2d0 P 834a3fe97ddf4cb5acb480d1384e8343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.420656 5565 raft_consensus.cc:374] T e114c23269424323bd004532a0c2d2d0 P 834a3fe97ddf4cb5acb480d1384e8343 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:52.420687 5565 raft_consensus.cc:729] T e114c23269424323bd004532a0c2d2d0 P 834a3fe97ddf4cb5acb480d1384e8343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 834a3fe97ddf4cb5acb480d1384e8343, State: Initialized, Role: FOLLOWER
I20260924 07:05:52.420800 5565 consensus_queue.cc:260] T e114c23269424323bd004532a0c2d2d0 P 834a3fe97ddf4cb5acb480d1384e8343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.421033 5565 ts_tablet_manager.cc:1428] T e114c23269424323bd004532a0c2d2d0 P 834a3fe97ddf4cb5acb480d1384e8343: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:05:52.421018 5571 tablet_replica.cc:439] T e114c23269424323bd004532a0c2d2d0 P 834a3fe97ddf4cb5acb480d1384e8343: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } } }
I20260924 07:05:52.421146 5571 tablet_replica.cc:442] T e114c23269424323bd004532a0c2d2d0 P 834a3fe97ddf4cb5acb480d1384e8343: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:05:52.588955 5573 raft_consensus.cc:482] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:05:52.589100 5573 raft_consensus.cc:504] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.590211 5573 leader_election.cc:288] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bbe530d020145e1b47b5e84f0ce5f39 (127.3.217.1:35761), 834a3fe97ddf4cb5acb480d1384e8343 (127.3.217.2:39199)
I20260924 07:05:52.593783 5560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:52.590550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:49379 (local address 127.3.217.1:35761)
0924 07:05:52.590662 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:52.590666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:52.590774 (+ 108us) server_negotiation.cc:407] Connection header received
0924 07:05:52.590987 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:52.590990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:52.591034 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:52.591095 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:05:52.591342 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.591885 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.592926 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.593308 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.593407 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:52.593468 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:52.593612 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 07:05:52.593684 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:05:52.593796 5605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:52.590460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:35761 (local address 127.3.217.3:49379)
0924 07:05:52.590693 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:52.590707 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:52.590812 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:52.591120 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:52.591123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:52.591139 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:05:52.591237 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:52.591242 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.591995 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.591998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:52.592799 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:52.592808 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.593415 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.593418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:52.593505 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:52.593524 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 07:05:52.593585 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 07:05:52.593652 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":63,"threads_started":1}
I20260924 07:05:52.593955 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:52.590907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:41009 (local address 127.3.217.2:39199)
0924 07:05:52.591013 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:52.591017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:52.591077 (+ 60us) server_negotiation.cc:407] Connection header received
0924 07:05:52.591218 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:52.591221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:52.591268 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:52.591327 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:05:52.591580 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.592084 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.593122 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.593490 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.593575 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:52.593629 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:52.593791 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 07:05:52.593849 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:05:52.593971 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:52.590779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:39199 (local address 127.3.217.3:41009)
0924 07:05:52.591002 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:52.591015 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:52.591119 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:52.591345 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:52.591348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:52.591363 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:05:52.591459 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:52.591465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.592182 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.592186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:52.593005 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:52.593014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.593591 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.593594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:52.593682 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:52.593702 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 07:05:52.593771 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 07:05:52.593836 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":55,"threads_started":1}
I20260924 07:05:52.594318 5216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e114c23269424323bd004532a0c2d2d0" candidate_uuid: "626aa2ff5fba421592e83f48f4fec2ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" is_pre_election: true
I20260924 07:05:52.594521 5216 raft_consensus.cc:2454] T e114c23269424323bd004532a0c2d2d0 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 626aa2ff5fba421592e83f48f4fec2ac in term 0.
I20260924 07:05:52.594502 5356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e114c23269424323bd004532a0c2d2d0" candidate_uuid: "626aa2ff5fba421592e83f48f4fec2ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "834a3fe97ddf4cb5acb480d1384e8343" is_pre_election: true
I20260924 07:05:52.594628 5356 raft_consensus.cc:2454] T e114c23269424323bd004532a0c2d2d0 P 834a3fe97ddf4cb5acb480d1384e8343 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 626aa2ff5fba421592e83f48f4fec2ac in term 0.
I20260924 07:05:52.594861 5427 leader_election.cc:302] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bbe530d020145e1b47b5e84f0ce5f39, 626aa2ff5fba421592e83f48f4fec2ac; no voters:
I20260924 07:05:52.595073 5573 raft_consensus.cc:2781] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:05:52.595140 5573 raft_consensus.cc:482] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:05:52.595183 5573 raft_consensus.cc:3037] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:52.596215 5573 raft_consensus.cc:504] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.596501 5573 leader_election.cc:288] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac [CANDIDATE]: Term 1 election: Requested vote from peers 4bbe530d020145e1b47b5e84f0ce5f39 (127.3.217.1:35761), 834a3fe97ddf4cb5acb480d1384e8343 (127.3.217.2:39199)
I20260924 07:05:52.596824 5216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e114c23269424323bd004532a0c2d2d0" candidate_uuid: "626aa2ff5fba421592e83f48f4fec2ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bbe530d020145e1b47b5e84f0ce5f39"
I20260924 07:05:52.596840 5356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e114c23269424323bd004532a0c2d2d0" candidate_uuid: "626aa2ff5fba421592e83f48f4fec2ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "834a3fe97ddf4cb5acb480d1384e8343"
I20260924 07:05:52.596936 5216 raft_consensus.cc:3037] T e114c23269424323bd004532a0c2d2d0 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:52.596959 5356 raft_consensus.cc:3037] T e114c23269424323bd004532a0c2d2d0 P 834a3fe97ddf4cb5acb480d1384e8343 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:52.597793 5216 raft_consensus.cc:2454] T e114c23269424323bd004532a0c2d2d0 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 626aa2ff5fba421592e83f48f4fec2ac in term 1.
I20260924 07:05:52.597851 5356 raft_consensus.cc:2454] T e114c23269424323bd004532a0c2d2d0 P 834a3fe97ddf4cb5acb480d1384e8343 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 626aa2ff5fba421592e83f48f4fec2ac in term 1.
I20260924 07:05:52.598318 5427 leader_election.cc:302] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 626aa2ff5fba421592e83f48f4fec2ac, 834a3fe97ddf4cb5acb480d1384e8343; no voters:
I20260924 07:05:52.598578 5573 raft_consensus.cc:2781] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:05:52.598740 5573 raft_consensus.cc:686] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac [term 1 LEADER]: Becoming Leader. State: Replica: 626aa2ff5fba421592e83f48f4fec2ac, State: Running, Role: LEADER
I20260924 07:05:52.598953 5573 consensus_queue.cc:237] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } }
I20260924 07:05:52.600191 5607 tablet_replica.cc:439] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac: TxnStatusTablet state changed. Reason: New leader 626aa2ff5fba421592e83f48f4fec2ac. Latest consensus state: current_term: 1 leader_uuid: "626aa2ff5fba421592e83f48f4fec2ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } } }
I20260924 07:05:52.600288 5607 tablet_replica.cc:442] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:05:52.600530 5609 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 07:05:52.600672 5609 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 07:05:52.600950 5059 catalog_manager.cc:5495] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 626aa2ff5fba421592e83f48f4fec2ac (127.3.217.3). New cstate: current_term: 1 leader_uuid: "626aa2ff5fba421592e83f48f4fec2ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } health_report { overall_health: HEALTHY } } }
I20260924 07:05:52.653223 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:52.649865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54926 (local address 127.3.217.3:36953)
0924 07:05:52.649986 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:52.649991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:52.650021 (+ 30us) server_negotiation.cc:407] Connection header received
0924 07:05:52.650215 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:52.650218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:52.650270 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:52.650347 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:05:52.650575 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.651065 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.651904 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.652110 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.652195 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:52.652463 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:05:52.652836 (+ 373us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:05:52.652931 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:52.653012 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 07:05:52.653060 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:05:52.653203 5601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:52.649747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:36953 (local address 127.0.0.1:54926)
0924 07:05:52.649966 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:52.649980 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:52.650083 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:52.650372 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:52.650375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:52.650383 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:05:52.650483 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:52.650488 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.651162 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.651165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:52.651798 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:52.651806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.652213 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.652216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:52.652301 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:52.652341 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:05:52.652927 (+ 586us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:05:52.652942 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 07:05:52.653003 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 07:05:52.653078 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260924 07:05:52.656024 5356 raft_consensus.cc:1261] T e114c23269424323bd004532a0c2d2d0 P 834a3fe97ddf4cb5acb480d1384e8343 [term 1 FOLLOWER]: Refusing update from remote peer 626aa2ff5fba421592e83f48f4fec2ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:05:52.656105 5216 raft_consensus.cc:1261] T e114c23269424323bd004532a0c2d2d0 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 1 FOLLOWER]: Refusing update from remote peer 626aa2ff5fba421592e83f48f4fec2ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:05:52.656510 5573 consensus_queue.cc:1035] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:05:52.656697 5607 consensus_queue.cc:1035] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:05:52.659008 5576 tablet_replica.cc:439] T e114c23269424323bd004532a0c2d2d0 P 4bbe530d020145e1b47b5e84f0ce5f39: TxnStatusTablet state changed. Reason: New leader 626aa2ff5fba421592e83f48f4fec2ac. Latest consensus state: current_term: 1 leader_uuid: "626aa2ff5fba421592e83f48f4fec2ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } } }
I20260924 07:05:52.659121 5576 tablet_replica.cc:442] T e114c23269424323bd004532a0c2d2d0 P 4bbe530d020145e1b47b5e84f0ce5f39: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:05:52.659822 5571 tablet_replica.cc:439] T e114c23269424323bd004532a0c2d2d0 P 834a3fe97ddf4cb5acb480d1384e8343: TxnStatusTablet state changed. Reason: New leader 626aa2ff5fba421592e83f48f4fec2ac. Latest consensus state: current_term: 1 leader_uuid: "626aa2ff5fba421592e83f48f4fec2ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } } }
I20260924 07:05:52.659929 5571 tablet_replica.cc:442] T e114c23269424323bd004532a0c2d2d0 P 834a3fe97ddf4cb5acb480d1384e8343: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:05:52.660725 5607 tablet_replica.cc:439] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "626aa2ff5fba421592e83f48f4fec2ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } } }
I20260924 07:05:52.660840 5607 tablet_replica.cc:442] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:05:52.661437 5573 tablet_replica.cc:439] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "626aa2ff5fba421592e83f48f4fec2ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } } }
I20260924 07:05:52.661532 5573 tablet_replica.cc:442] T e114c23269424323bd004532a0c2d2d0 P 626aa2ff5fba421592e83f48f4fec2ac: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:05:52.665416 5576 tablet_replica.cc:439] T e114c23269424323bd004532a0c2d2d0 P 4bbe530d020145e1b47b5e84f0ce5f39: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "626aa2ff5fba421592e83f48f4fec2ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } } }
I20260924 07:05:52.665525 5576 tablet_replica.cc:442] T e114c23269424323bd004532a0c2d2d0 P 4bbe530d020145e1b47b5e84f0ce5f39: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:05:52.666009 5579 mvcc.cc:205] Tried to move back new op lower bound from 7332796631673921536 to 7332796631447375872. Current Snapshot: MvccSnapshot[applied={T|T < 7332796631673921536 or (T in {7332796631673921536})}]
I20260924 07:05:52.667840 5571 tablet_replica.cc:439] T e114c23269424323bd004532a0c2d2d0 P 834a3fe97ddf4cb5acb480d1384e8343: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "626aa2ff5fba421592e83f48f4fec2ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbe530d020145e1b47b5e84f0ce5f39" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35761 } } peers { permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 } } peers { permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 } } }
I20260924 07:05:52.667943 5571 tablet_replica.cc:442] T e114c23269424323bd004532a0c2d2d0 P 834a3fe97ddf4cb5acb480d1384e8343: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:05:52.675467 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:52.672345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:48157 (local address 127.3.217.3:36953)
0924 07:05:52.672456 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:52.672460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:52.672473 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:05:52.672657 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:52.672659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:52.672714 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:52.672772 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:05:52.673034 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.673758 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.674555 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.674768 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.674850 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:52.675090 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:05:52.675165 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:05:52.675221 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:52.675307 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 07:05:52.675350 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:05:52.675523 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:52.672090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:36953 (local address 127.3.217.1:48157)
0924 07:05:52.672372 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:52.672388 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:52.672477 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:52.672824 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:52.672828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:52.672837 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:05:52.672949 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:52.672955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.673860 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.673864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:52.674457 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:52.674465 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.674857 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.674861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:52.674943 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:52.674982 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:05:52.675235 (+ 253us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:05:52.675241 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:05:52.675296 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 07:05:52.675375 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":75,"threads_started":1}
I20260924 07:05:52.681507 5356 raft_consensus.cc:1261] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 834a3fe97ddf4cb5acb480d1384e8343 [term 1 FOLLOWER]: Refusing update from remote peer 4bbe530d020145e1b47b5e84f0ce5f39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:05:52.681521 5496 raft_consensus.cc:1261] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 626aa2ff5fba421592e83f48f4fec2ac [term 1 FOLLOWER]: Refusing update from remote peer 4bbe530d020145e1b47b5e84f0ce5f39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:05:52.681891 5585 consensus_queue.cc:1035] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "834a3fe97ddf4cb5acb480d1384e8343" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:05:52.682044 5576 consensus_queue.cc:1035] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "626aa2ff5fba421592e83f48f4fec2ac" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:05:52.701440 5621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:52.697694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:35761 (local address 127.3.217.3:55175)
0924 07:05:52.697937 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:52.697950 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:52.698019 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:52.698300 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:52.698303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:52.698310 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:05:52.698391 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:52.698395 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.699398 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.699401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:52.700073 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:52.700082 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.700695 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.700699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:52.700770 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:52.700808 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:05:52.701168 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:05:52.701175 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 07:05:52.701241 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 07:05:52.701305 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":63,"threads_started":1}
I20260924 07:05:52.701445 5560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:52.697879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:55175 (local address 127.3.217.1:35761)
0924 07:05:52.697990 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:52.697994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:52.698008 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:05:52.698086 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:52.698089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:52.698133 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:52.698207 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:05:52.698509 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.699265 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.700188 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.700497 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.700606 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:52.700939 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:05:52.701021 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:05:52.701110 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:52.701266 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 07:05:52.701321 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:05:52.722285 5058 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:05:52.722455 5058 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:05:52.724761 5058 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c91b156a1f149a08cce4ab8c45e1753: Sending DeleteTablet for 3 replicas of tablet bb7f9b2bfc4449b5ab44ebf970bdead0
I20260924 07:05:52.725317 5196 tablet_service.cc:1514] Processing DeleteTablet for tablet bb7f9b2bfc4449b5ab44ebf970bdead0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:05:52 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:38394
I20260924 07:05:52.725492 5336 tablet_service.cc:1514] Processing DeleteTablet for tablet bb7f9b2bfc4449b5ab44ebf970bdead0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:05:52 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:43936
I20260924 07:05:52.725674 5589 tablet_replica.cc:330] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39: stopping tablet replica
I20260924 07:05:52.725793 5589 raft_consensus.cc:2229] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:05:52.725956 5589 raft_consensus.cc:2258] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:05:52.726114 5591 tablet_replica.cc:330] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 834a3fe97ddf4cb5acb480d1384e8343: stopping tablet replica
I20260924 07:05:52.726217 5591 raft_consensus.cc:2229] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 834a3fe97ddf4cb5acb480d1384e8343 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:05:52.726338 5591 raft_consensus.cc:2258] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 834a3fe97ddf4cb5acb480d1384e8343 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:05:52.726742 5476 tablet_service.cc:1514] Processing DeleteTablet for tablet bb7f9b2bfc4449b5ab44ebf970bdead0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:05:52 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:54912
I20260924 07:05:52.727053 5589 ts_tablet_manager.cc:1905] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:05:52.727135 5591 ts_tablet_manager.cc:1905] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 834a3fe97ddf4cb5acb480d1384e8343: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:05:52.729285 5589 ts_tablet_manager.cc:1918] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:05:52.729350 5589 log.cc:1196] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-0/wal/wals/bb7f9b2bfc4449b5ab44ebf970bdead0
I20260924 07:05:52.729548 5589 ts_tablet_manager.cc:1939] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 4bbe530d020145e1b47b5e84f0ce5f39: Deleting consensus metadata
I20260924 07:05:52.729734 5590 tablet_replica.cc:330] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 626aa2ff5fba421592e83f48f4fec2ac: stopping tablet replica
I20260924 07:05:52.729858 5590 raft_consensus.cc:2229] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 626aa2ff5fba421592e83f48f4fec2ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:05:52.729907 5591 ts_tablet_manager.cc:1918] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 834a3fe97ddf4cb5acb480d1384e8343: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:05:52.729946 5590 raft_consensus.cc:2258] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 626aa2ff5fba421592e83f48f4fec2ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:05:52.729981 5591 log.cc:1196] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 834a3fe97ddf4cb5acb480d1384e8343: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/bb7f9b2bfc4449b5ab44ebf970bdead0
I20260924 07:05:52.730729 5590 ts_tablet_manager.cc:1905] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 626aa2ff5fba421592e83f48f4fec2ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:05:52.733512 5590 ts_tablet_manager.cc:1918] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 626aa2ff5fba421592e83f48f4fec2ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:05:52.733593 5590 log.cc:1196] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 626aa2ff5fba421592e83f48f4fec2ac: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/bb7f9b2bfc4449b5ab44ebf970bdead0
I20260924 07:05:52.734203 5043 catalog_manager.cc:4841] TS 4bbe530d020145e1b47b5e84f0ce5f39 (127.3.217.1:35761): tablet bb7f9b2bfc4449b5ab44ebf970bdead0 (table test-table [id=bd4c1857d93e428eab02848714715d9c]) successfully deleted
I20260924 07:05:52.734366 5590 ts_tablet_manager.cc:1939] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 626aa2ff5fba421592e83f48f4fec2ac: Deleting consensus metadata
I20260924 07:05:52.735114 5044 catalog_manager.cc:4841] TS 626aa2ff5fba421592e83f48f4fec2ac (127.3.217.3:36953): tablet bb7f9b2bfc4449b5ab44ebf970bdead0 (table test-table [id=bd4c1857d93e428eab02848714715d9c]) successfully deleted
I20260924 07:05:52.735549 5591 ts_tablet_manager.cc:1939] T bb7f9b2bfc4449b5ab44ebf970bdead0 P 834a3fe97ddf4cb5acb480d1384e8343: Deleting consensus metadata
Sep 24 07:05:52 dist-test-slave-2p7j krb5kdc[5014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233551, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.217.1@KRBTEST.COM
I20260924 07:05:52.736429 5043 catalog_manager.cc:4841] TS 834a3fe97ddf4cb5acb480d1384e8343 (127.3.217.2:39199): tablet bb7f9b2bfc4449b5ab44ebf970bdead0 (table test-table [id=bd4c1857d93e428eab02848714715d9c]) successfully deleted
I20260924 07:05:52.741218 5560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:52.730323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38412 (local address 127.3.217.1:35761)
0924 07:05:52.730433 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:52.730438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:52.730746 (+ 308us) server_negotiation.cc:407] Connection header received
0924 07:05:52.730883 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:52.730886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:52.730931 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:52.730992 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:52.731661 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.732184 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.733205 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.733507 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.733579 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:52.736884 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:52.736894 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:52.736908 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:52.736931 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:52.738825 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:52.739679 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:52.739685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:52.739688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:52.739780 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:52.740100 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:52.740104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:52.740106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:52.740396 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:52.740646 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:52.740953 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 07:05:52.741064 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 07:05:52.741662 5256 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38412
I20260924 07:05:52.751260 5108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:52.743103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.3.217.62:36735)
0924 07:05:52.743201 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:52.743205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:52.743645 (+ 440us) server_negotiation.cc:407] Connection header received
0924 07:05:52.743739 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:52.743741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:52.743787 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:52.743862 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:52.744572 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.745370 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.746452 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:52.746784 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:52.746880 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:52.748278 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:52.748281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:52.748284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:52.748307 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:52.749813 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:52.750207 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:52.750210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:52.750212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:52.750290 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:52.750512 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:52.750515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:52.750517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:52.750697 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:52.750819 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:52.751013 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 07:05:52.751122 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 07:05:52.751645 5058 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59434
I20260924 07:05:52.752645 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 5131
I20260924 07:05:52.761947 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 5271
I20260924 07:05:52.766953 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 5411
I20260924 07:05:52.772814 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 5027
2026-09-24T07:05:52Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3986 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:05:52 dist-test-slave-2p7j krb5kdc[5629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:05:52 dist-test-slave-2p7j krb5kdc[5629](info): set up 2 sockets
Sep 24 07:05:52 dist-test-slave-2p7j krb5kdc[5629](info): commencing operation
krb5kdc: starting...
W20260924 07:05:53.755542 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:05:53 dist-test-slave-2p7j krb5kdc[5629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:05:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:05:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:05:53.872157 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:33557
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:40545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:33557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:05:53.973758 5642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:05:53.973991 5642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:05:53.974041 5642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:05:53.977890 5642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:05:53.977938 5642 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 07:05:53.977960 5642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:05:53.977983 5642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:05:53.978004 5642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:05:53.978032 5642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:05:53.982972 5642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:40545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:33557
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:33557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:05:53.983932 5642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:05:53.984836 5642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:05:53.993170 5650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:53.993279 5648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:05:53.993392 5642 server_base.cc:1034] running on GCE node
W20260924 07:05:53.993136 5647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:05:53.994056 5642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:05:53.995173 5642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:05:53.996351 5642 hybrid_clock.cc:648] HybridClock initialized: now 1790233553996317 us; error 43 us; skew 500 ppm
Sep 24 07:05:53 dist-test-slave-2p7j krb5kdc[5629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233553, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:05:53.998952 5642 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:05:53.999804 5642 webserver.cc:460] Webserver started at http://127.3.217.62:46157/ using document root <none> and password file <none>
I20260924 07:05:54.000289 5642 fs_manager.cc:359] Metadata directory not provided
I20260924 07:05:54.000351 5642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:05:54.000505 5642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:05:54.002421 5642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "7677ccc4700f44efa07ff885f44f8bc9"
format_stamp: "Formatted at 2026-09-24 07:05:53 on dist-test-slave-2p7j"
server_key: "dcee4ad28e98bb60e6ac2a25f61f07ee"
server_key_iv: "5e575ba11ba946eac02d960783d27bca"
server_key_version: "encryptionkey@0"
I20260924 07:05:54.002920 5642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "7677ccc4700f44efa07ff885f44f8bc9"
format_stamp: "Formatted at 2026-09-24 07:05:53 on dist-test-slave-2p7j"
server_key: "dcee4ad28e98bb60e6ac2a25f61f07ee"
server_key_iv: "5e575ba11ba946eac02d960783d27bca"
server_key_version: "encryptionkey@0"
I20260924 07:05:54.006501 5642 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 07:05:54.009084 5657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:54.010223 5642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:05:54.010341 5642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "7677ccc4700f44efa07ff885f44f8bc9"
format_stamp: "Formatted at 2026-09-24 07:05:53 on dist-test-slave-2p7j"
server_key: "dcee4ad28e98bb60e6ac2a25f61f07ee"
server_key_iv: "5e575ba11ba946eac02d960783d27bca"
server_key_version: "encryptionkey@0"
I20260924 07:05:54.010423 5642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:05:54.029861 5642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:05:54.033222 5642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:05:54.033388 5642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:05:54.041399 5642 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:33557
I20260924 07:05:54.041424 5719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:33557 every 8 connection(s)
I20260924 07:05:54.042260 5642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:05:54.044893 5720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:54.047186 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 5642
I20260924 07:05:54.047303 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:05:54.047596 3940 external_mini_cluster.cc:1428] Setting key f6c460f8a4b2914acc86000fdc352dc4
I20260924 07:05:54.050475 5720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9: Bootstrap starting.
I20260924 07:05:54.052717 5720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:54.053525 5720 log.cc:824] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9: Log is configured to *not* fsync() on all Append() calls
I20260924 07:05:54.055526 5720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9: No bootstrap required, opened a new log
Sep 24 07:05:54 dist-test-slave-2p7j krb5kdc[5629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:05:54.058969 5720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7677ccc4700f44efa07ff885f44f8bc9" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33557 } }
I20260924 07:05:54.059183 5720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:54.059221 5720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7677ccc4700f44efa07ff885f44f8bc9, State: Initialized, Role: FOLLOWER
I20260924 07:05:54.059682 5720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7677ccc4700f44efa07ff885f44f8bc9" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33557 } }
I20260924 07:05:54.059840 5720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:05:54.059949 5720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:05:54.060050 5720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:54.060809 5720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7677ccc4700f44efa07ff885f44f8bc9" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33557 } }
I20260924 07:05:54.061110 5720 leader_election.cc:302] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7677ccc4700f44efa07ff885f44f8bc9; no voters:
I20260924 07:05:54.061372 5720 leader_election.cc:288] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:05:54.061473 5725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:05:54.061681 5725 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9 [term 1 LEADER]: Becoming Leader. State: Replica: 7677ccc4700f44efa07ff885f44f8bc9, State: Running, Role: LEADER
I20260924 07:05:54.062027 5725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7677ccc4700f44efa07ff885f44f8bc9" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33557 } }
I20260924 07:05:54.062461 5720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:05:54.062470 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.049238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38064 (local address 127.3.217.62:33557)
0924 07:05:54.049713 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:54.049727 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:54.049763 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:05:54.050355 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:54.050370 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:54.050696 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:54.050983 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:54.051862 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.052844 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.053930 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.054483 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.054593 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.057397 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:54.057405 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:54.057426 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:54.057472 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:54.059457 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:54.059939 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:54.059945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:54.059950 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:54.060078 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:54.060350 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:54.060354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:54.060356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:54.060708 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:54.060899 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:54.061136 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 07:05:54.061569 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":143,"threads_started":1}
I20260924 07:05:54.063131 5727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7677ccc4700f44efa07ff885f44f8bc9. Latest consensus state: current_term: 1 leader_uuid: "7677ccc4700f44efa07ff885f44f8bc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7677ccc4700f44efa07ff885f44f8bc9" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33557 } } }
I20260924 07:05:54.063868 5727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9 [sys.catalog]: This master's current role is: LEADER
I20260924 07:05:54.063756 5726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7677ccc4700f44efa07ff885f44f8bc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7677ccc4700f44efa07ff885f44f8bc9" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33557 } } }
I20260924 07:05:54.064179 5726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9 [sys.catalog]: This master's current role is: LEADER
W20260924 07:05:54.067101 5741 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:05:54.067214 5741 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:05:54.067432 5734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:05:54.068126 5734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:05:54.072540 5734 catalog_manager.cc:1347] Generated new cluster ID: abe1d5c838694d7ab5c5ae3479d7035e
I20260924 07:05:54.072598 5734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:05:54.085004 5734 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:05:54.085845 5734 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:05:54.093250 5734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9: Generated new TSK 0
I20260924 07:05:54.093895 5734 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
I20260924 07:05:54.126586 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:33557
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:40545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:05:54.239493 5746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:05:54.239766 5746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:05:54.239817 5746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:05:54.243810 5746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:05:54.243863 5746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:05:54.243906 5746 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 07:05:54.243937 5746 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:05:54.243993 5746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:05:54.248996 5746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:40545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.217.62:33557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:05:54.249943 5746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:05:54.250931 5746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:05:54.261894 5752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:54.261919 5751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:54.261965 5754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:05:54.262087 5746 server_base.cc:1034] running on GCE node
I20260924 07:05:54.262621 5746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:05:54.263293 5746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:05:54.264463 5746 hybrid_clock.cc:648] HybridClock initialized: now 1790233554264441 us; error 36 us; skew 500 ppm
Sep 24 07:05:54 dist-test-slave-2p7j krb5kdc[5629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233554, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:05:54.267445 5746 init.cc:377] Logged in from keytab as kudu/127.3.217.1@KRBTEST.COM (short username kudu)
I20260924 07:05:54.268304 5746 webserver.cc:460] Webserver started at http://127.3.217.1:37585/ using document root <none> and password file <none>
I20260924 07:05:54.268817 5746 fs_manager.cc:359] Metadata directory not provided
I20260924 07:05:54.268875 5746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:05:54.269035 5746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:05:54.270888 5746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "19459ca02e324f3c9492133944ba2763"
format_stamp: "Formatted at 2026-09-24 07:05:54 on dist-test-slave-2p7j"
server_key: "56e5653a3eeb565847ac77e40a5bb4e0"
server_key_iv: "59b35f244749f786960880676ae55c29"
server_key_version: "encryptionkey@0"
I20260924 07:05:54.271376 5746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "19459ca02e324f3c9492133944ba2763"
format_stamp: "Formatted at 2026-09-24 07:05:54 on dist-test-slave-2p7j"
server_key: "56e5653a3eeb565847ac77e40a5bb4e0"
server_key_iv: "59b35f244749f786960880676ae55c29"
server_key_version: "encryptionkey@0"
I20260924 07:05:54.274745 5746 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260924 07:05:54.277077 5761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:54.278291 5746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:05:54.278400 5746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "19459ca02e324f3c9492133944ba2763"
format_stamp: "Formatted at 2026-09-24 07:05:54 on dist-test-slave-2p7j"
server_key: "56e5653a3eeb565847ac77e40a5bb4e0"
server_key_iv: "59b35f244749f786960880676ae55c29"
server_key_version: "encryptionkey@0"
I20260924 07:05:54.278496 5746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:05:54.310464 5746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:05:54.313532 5746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:05:54.313714 5746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:05:54.315188 5746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:05:54.315241 5746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:54.315299 5746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:05:54.315331 5746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:05:54 dist-test-slave-2p7j krb5kdc[5629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233554, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:05:54.326251 5746 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:42453
I20260924 07:05:54.326313 5876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:42453 every 8 connection(s)
I20260924 07:05:54.327095 5746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:05:54.328644 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.315943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:47367 (local address 127.3.217.62:33557)
0924 07:05:54.316092 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:54.316097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:54.316921 (+ 824us) server_negotiation.cc:407] Connection header received
0924 07:05:54.318092 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:54.318096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:54.318148 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:54.318252 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:54.319775 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.320448 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.321605 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.321877 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.321996 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.325226 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:54.325230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:54.325232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:54.325258 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:54.326654 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:54.327093 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:54.327097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:54.327099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:54.327169 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:54.327559 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:54.327563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:54.327565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:54.327765 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:54.327899 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:54.328408 (+ 509us) server_negotiation.cc:299] Negotiation successful
0924 07:05:54.328523 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:05:54.329309 5770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.316295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:33557 (local address 127.3.217.1:47367)
0924 07:05:54.316750 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:54.316806 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:54.317854 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:54.318426 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:54.318435 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:54.318909 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:05:54.319609 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:54.319638 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.320593 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.320596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:54.321493 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:54.321502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.322103 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.322110 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:54.322262 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.322808 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:05:54.322825 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:05:54.325082 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:05:54.326752 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:54.326757 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:54.326770 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:54.327005 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:54.327253 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:54.327257 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:54.327259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:54.327433 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:54.327900 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:05:54.327906 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:05:54.328159 (+ 253us) client_negotiation.cc:769] Sending connection context
0924 07:05:54.328310 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 07:05:54.328535 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":94,"threads_started":1}
I20260924 07:05:54.330828 5877 heartbeater.cc:347] Connected to a master server at 127.3.217.62:33557
I20260924 07:05:54.330976 5877 heartbeater.cc:464] Registering TS with master...
I20260924 07:05:54.331421 5877 heartbeater.cc:511] Master 127.3.217.62:33557 requested a full tablet report, sending...
I20260924 07:05:54.332230 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 5746
I20260924 07:05:54.332376 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:05:54.332727 3940 external_mini_cluster.cc:1428] Setting key 7ccf4f1014c17c726d865dce20719eca
I20260924 07:05:54.332852 5673 ts_manager.cc:194] Registered new tserver with Master: 19459ca02e324f3c9492133944ba2763 (127.3.217.1:42453)
I20260924 07:05:54.334205 5673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.1@KRBTEST.COM'} at 127.3.217.1:47367
I20260924 07:05:54.341760 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.333867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:48805 (local address 127.3.217.62:33557)
0924 07:05:54.334013 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:54.334017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:54.334146 (+ 129us) server_negotiation.cc:407] Connection header received
0924 07:05:54.334308 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:54.334311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:54.334361 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:54.334432 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:54.334957 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.335757 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.336744 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.337089 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.337156 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.338401 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:54.338404 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:54.338406 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:54.338426 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:54.340050 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:54.340562 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:54.340564 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:54.340565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:54.340637 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:54.341037 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:54.341039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:54.341040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:54.341217 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:54.341304 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:54.341563 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 07:05:54.341651 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 07:05:54.341866 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.333611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:33557 (local address 127.3.217.1:48805)
0924 07:05:54.334038 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:54.334050 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:54.334147 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:54.334536 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:54.334539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:54.334710 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:05:54.334819 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:54.334827 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.335883 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.335887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:54.336613 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:54.336623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.337307 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.337311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:54.337398 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.337861 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:05:54.337874 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:05:54.338239 (+ 365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:05:54.340192 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:54.340196 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:54.340198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:54.340453 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:54.340725 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:54.340728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:54.340730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:54.340828 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:54.341314 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:05:54.341317 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:05:54.341412 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 07:05:54.341527 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 07:05:54.341662 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":2,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.3.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
I20260924 07:05:54.361905 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:33557
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:40545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:05:54.469640 5886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:05:54.469864 5886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:05:54.469936 5886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:05:54.473841 5886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:05:54.473894 5886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:05:54.473949 5886 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 07:05:54.473971 5886 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:05:54.474016 5886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:05:54.478942 5886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:40545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.217.62:33557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:05:54.479851 5886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:05:54.480742 5886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:05:54.490353 5891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:54.490381 5894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:54.490425 5892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:05:54.490561 5886 server_base.cc:1034] running on GCE node
I20260924 07:05:54.490916 5886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:05:54.491480 5886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:05:54.492803 5886 hybrid_clock.cc:648] HybridClock initialized: now 1790233554492727 us; error 56 us; skew 500 ppm
Sep 24 07:05:54 dist-test-slave-2p7j krb5kdc[5629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233554, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:05:54.495486 5886 init.cc:377] Logged in from keytab as kudu/127.3.217.2@KRBTEST.COM (short username kudu)
I20260924 07:05:54.496302 5886 webserver.cc:460] Webserver started at http://127.3.217.2:42599/ using document root <none> and password file <none>
I20260924 07:05:54.496814 5886 fs_manager.cc:359] Metadata directory not provided
I20260924 07:05:54.496879 5886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:05:54.497069 5886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:05:54.499977 5886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "96f31cc614d5499293eafa649dd9eb6c"
format_stamp: "Formatted at 2026-09-24 07:05:54 on dist-test-slave-2p7j"
server_key: "17d1bea7cc7800095aa134e9d402d965"
server_key_iv: "9ee0a8bc3ca5da64bd2d77d14f330e60"
server_key_version: "encryptionkey@0"
I20260924 07:05:54.500752 5886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "96f31cc614d5499293eafa649dd9eb6c"
format_stamp: "Formatted at 2026-09-24 07:05:54 on dist-test-slave-2p7j"
server_key: "17d1bea7cc7800095aa134e9d402d965"
server_key_iv: "9ee0a8bc3ca5da64bd2d77d14f330e60"
server_key_version: "encryptionkey@0"
I20260924 07:05:54.504758 5886 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 07:05:54.507225 5901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:54.508471 5886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260924 07:05:54.508571 5886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "96f31cc614d5499293eafa649dd9eb6c"
format_stamp: "Formatted at 2026-09-24 07:05:54 on dist-test-slave-2p7j"
server_key: "17d1bea7cc7800095aa134e9d402d965"
server_key_iv: "9ee0a8bc3ca5da64bd2d77d14f330e60"
server_key_version: "encryptionkey@0"
I20260924 07:05:54.508764 5886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:05:54.518785 5886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:05:54.521477 5886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:05:54.521625 5886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:05:54.522951 5886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:05:54.522998 5886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:54.523047 5886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:05:54.523074 5886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:05:54 dist-test-slave-2p7j krb5kdc[5629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233554, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:05:54.532655 5886 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:40169
I20260924 07:05:54.532716 6016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:40169 every 8 connection(s)
I20260924 07:05:54.533579 5886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:05:54.536259 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.524192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:33933 (local address 127.3.217.62:33557)
0924 07:05:54.524312 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:54.524316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:54.525556 (+ 1240us) server_negotiation.cc:407] Connection header received
0924 07:05:54.526705 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:54.526709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:54.526761 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:54.526853 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:54.528143 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.528917 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.530102 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.530331 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.530413 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.532888 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:54.532892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:54.532895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:54.532918 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:54.534467 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:54.534918 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:54.534920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:54.534921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:54.534989 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:54.535314 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:54.535317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:54.535319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:54.535494 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:54.535602 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:54.536038 (+ 436us) server_negotiation.cc:299] Negotiation successful
0924 07:05:54.536128 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:05:54.536991 5910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.524638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:33557 (local address 127.3.217.2:33933)
0924 07:05:54.525278 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:54.525357 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:54.526489 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:54.527032 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:54.527040 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:54.527467 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:05:54.528016 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:54.528028 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.529034 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.529037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:54.530002 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:54.530012 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.530455 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.530459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:54.530619 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.531204 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:05:54.531222 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:05:54.532745 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:05:54.534564 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:54.534569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:54.534578 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:54.534835 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:54.535086 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:54.535088 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:54.535089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:54.535226 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:54.535604 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:05:54.535608 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:05:54.535828 (+ 220us) client_negotiation.cc:769] Sending connection context
0924 07:05:54.535985 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 07:05:54.536222 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":115,"threads_started":1}
I20260924 07:05:54.538280 6017 heartbeater.cc:347] Connected to a master server at 127.3.217.62:33557
I20260924 07:05:54.538411 6017 heartbeater.cc:464] Registering TS with master...
I20260924 07:05:54.538573 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 5886
I20260924 07:05:54.538693 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:05:54.538923 6017 heartbeater.cc:511] Master 127.3.217.62:33557 requested a full tablet report, sending...
I20260924 07:05:54.539011 3940 external_mini_cluster.cc:1428] Setting key 3dfb948de6522a23708b1ec3fe28f34f
I20260924 07:05:54.539892 5674 ts_manager.cc:194] Registered new tserver with Master: 96f31cc614d5499293eafa649dd9eb6c (127.3.217.2:40169)
I20260924 07:05:54.540560 5674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.2@KRBTEST.COM'} at 127.3.217.2:33933
I20260924 07:05:54.547327 6023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.540420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:33557 (local address 127.3.217.2:45965)
0924 07:05:54.540746 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:54.540759 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:54.540853 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:54.541072 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:54.541075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:54.541289 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:05:54.541403 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:54.541411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.542369 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.542372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:54.543069 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:54.543078 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.543636 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.543639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:54.543706 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.544106 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:05:54.544117 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:05:54.544401 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:05:54.546046 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:54.546048 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:54.546050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:54.546220 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:54.546471 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:54.546472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:54.546474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:54.546536 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:54.546954 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:05:54.546956 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:05:54.547025 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 07:05:54.547101 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 07:05:54.547193 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":76,"threads_started":1}
I20260924 07:05:54.547427 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.540493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:45965 (local address 127.3.217.62:33557)
0924 07:05:54.540600 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:54.540604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:54.540829 (+ 225us) server_negotiation.cc:407] Connection header received
0924 07:05:54.540926 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:54.540929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:54.540973 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:54.541047 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:54.541517 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.542235 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.543178 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.543534 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.543635 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.544524 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:54.544528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:54.544531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:54.544553 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:54.545939 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:54.546307 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:54.546310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:54.546312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:54.546395 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:54.546607 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:54.546610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:54.546612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:54.546865 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:54.546966 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:54.547171 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 07:05:54.547288 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for kudu/127.3.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
I20260924 07:05:54.567402 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:33557
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:40545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:05:54.680104 6026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:05:54.680591 6026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:05:54.680665 6026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:05:54.686707 6026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:05:54.686808 6026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:05:54.686887 6026 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 07:05:54.686924 6026 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:05:54.686996 6026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:05:54.694792 6026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:40545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.217.62:33557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:05:54.695839 6026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:05:54.696983 6026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:05:54.707053 6032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:05:54.707129 6026 server_base.cc:1034] running on GCE node
W20260924 07:05:54.707053 6031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:05:54.707291 6034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:05:54.707674 6026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:05:54.708282 6026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:05:54.709558 6026 hybrid_clock.cc:648] HybridClock initialized: now 1790233554709542 us; error 30 us; skew 500 ppm
Sep 24 07:05:54 dist-test-slave-2p7j krb5kdc[5629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233554, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:05:54.712262 6026 init.cc:377] Logged in from keytab as kudu/127.3.217.3@KRBTEST.COM (short username kudu)
I20260924 07:05:54.713053 6026 webserver.cc:460] Webserver started at http://127.3.217.3:46815/ using document root <none> and password file <none>
I20260924 07:05:54.713581 6026 fs_manager.cc:359] Metadata directory not provided
I20260924 07:05:54.713651 6026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:05:54.713868 6026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:05:54.715837 6026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "c6f818ff85da4dc497a5a05ab87e02b0"
format_stamp: "Formatted at 2026-09-24 07:05:54 on dist-test-slave-2p7j"
server_key: "73aefea410569c81041d426022a608bd"
server_key_iv: "43c6d007a59e13e8334f1578043bfad2"
server_key_version: "encryptionkey@0"
I20260924 07:05:54.716475 6026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "c6f818ff85da4dc497a5a05ab87e02b0"
format_stamp: "Formatted at 2026-09-24 07:05:54 on dist-test-slave-2p7j"
server_key: "73aefea410569c81041d426022a608bd"
server_key_iv: "43c6d007a59e13e8334f1578043bfad2"
server_key_version: "encryptionkey@0"
I20260924 07:05:54.720065 6026 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:05:54.722443 6041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:54.723654 6026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:05:54.723779 6026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "c6f818ff85da4dc497a5a05ab87e02b0"
format_stamp: "Formatted at 2026-09-24 07:05:54 on dist-test-slave-2p7j"
server_key: "73aefea410569c81041d426022a608bd"
server_key_iv: "43c6d007a59e13e8334f1578043bfad2"
server_key_version: "encryptionkey@0"
I20260924 07:05:54.723868 6026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:05:54.743892 6026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:05:54.746548 6026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:05:54.746702 6026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:05:54.748064 6026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:05:54.748121 6026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:05:54.748170 6026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:05:54.748199 6026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:05:54 dist-test-slave-2p7j krb5kdc[5629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233554, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:05:54.758287 6026 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:46651
I20260924 07:05:54.758530 6156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:46651 every 8 connection(s)
I20260924 07:05:54.759351 6026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:05:54.761430 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.749508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:37255 (local address 127.3.217.62:33557)
0924 07:05:54.749619 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:54.749622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:54.750604 (+ 982us) server_negotiation.cc:407] Connection header received
0924 07:05:54.751767 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:54.751771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:54.751828 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:54.751908 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:54.753096 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.753705 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.754893 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.755104 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.755201 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.757975 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:54.757979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:54.757982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:54.758007 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:54.759477 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:54.760041 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:54.760044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:54.760046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:54.760133 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:54.760456 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:54.760460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:54.760461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:54.760633 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:54.760715 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:54.761150 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 07:05:54.761264 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:05:54.762116 6050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.749931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:33557 (local address 127.3.217.3:37255)
0924 07:05:54.750457 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:54.750500 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:54.751537 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:54.752083 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:54.752092 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:54.752471 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:05:54.752951 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:54.752965 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.753824 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.753828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:54.754772 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:54.754782 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.755277 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.755282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:54.755471 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.756026 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:05:54.756043 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:05:54.757718 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:05:54.759602 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:54.759606 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:54.759616 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:54.759922 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:54.760217 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:54.760219 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:54.760222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:54.760368 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:54.760724 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:05:54.760730 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:05:54.760940 (+ 210us) client_negotiation.cc:769] Sending connection context
0924 07:05:54.761095 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 07:05:54.761378 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":124,"threads_started":1}
I20260924 07:05:54.762992 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 6026
I20260924 07:05:54.763111 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:05:54.763810 3940 external_mini_cluster.cc:1428] Setting key 5984d48e3a7cb6ab2e37684a088c2297
I20260924 07:05:54.764029 6157 heartbeater.cc:347] Connected to a master server at 127.3.217.62:33557
I20260924 07:05:54.764178 6157 heartbeater.cc:464] Registering TS with master...
I20260924 07:05:54.764714 6157 heartbeater.cc:511] Master 127.3.217.62:33557 requested a full tablet report, sending...
I20260924 07:05:54.765713 5673 ts_manager.cc:194] Registered new tserver with Master: c6f818ff85da4dc497a5a05ab87e02b0 (127.3.217.3:46651)
I20260924 07:05:54.766424 5673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.3@KRBTEST.COM'} at 127.3.217.3:37255
I20260924 07:05:54.773833 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.766773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:60885 (local address 127.3.217.62:33557)
0924 07:05:54.766875 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:54.766878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:54.767111 (+ 233us) server_negotiation.cc:407] Connection header received
0924 07:05:54.767236 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:54.767239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:54.767281 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:54.767442 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:54.767923 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.768407 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.769147 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.769364 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.769445 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.770798 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:54.770802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:54.770804 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:54.770833 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:54.772385 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:54.772814 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:54.772817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:54.772819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:54.772891 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:54.773135 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:54.773138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:54.773140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:54.773298 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:54.773379 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:54.773605 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 07:05:54.773698 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 07:05:54.773870 6163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.766810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:33557 (local address 127.3.217.3:60885)
0924 07:05:54.767033 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:54.767045 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:54.767156 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:54.767452 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:54.767455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:54.767675 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:05:54.767810 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:54.767817 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.768517 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.768521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:54.769049 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:54.769056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.769455 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.769459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:54.769548 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.770138 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:05:54.770155 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:05:54.770567 (+ 412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:05:54.772509 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:54.772513 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:54.772516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:54.772722 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:54.772975 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:54.772978 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:54.772980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:54.773059 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:54.773381 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:05:54.773384 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:05:54.773466 (+ 82us) client_negotiation.cc:769] Sending connection context
0924 07:05:54.773546 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 07:05:54.773676 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260924 07:05:54.778118 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 07:05:54 dist-test-slave-2p7j krb5kdc[5629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 07:05:54 dist-test-slave-2p7j krb5kdc[5629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:05:54.803984 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.795108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38072 (local address 127.3.217.62:33557)
0924 07:05:54.795221 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:54.795224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:54.795236 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:05:54.795274 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:54.795276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:54.795312 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:54.795383 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:54.796462 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.796971 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.798151 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.798456 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.798555 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.801052 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:54.801055 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:54.801057 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:54.801078 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:54.802458 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:54.802871 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:54.802873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:54.802874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:54.802937 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:54.803186 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:54.803189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:54.803190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:54.803366 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:54.803485 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:54.803780 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 07:05:54.803868 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:05:54.807176 5673 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:05:54.810134 5673 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:05:54.826021 6175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.819079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:46651 (local address 127.0.0.1:53284)
0924 07:05:54.820002 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:54.820016 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:54.820104 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:54.821014 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:54.821018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:54.821034 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:05:54.821117 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:54.821123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.822387 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.822390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:54.823972 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:54.823983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.825314 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.825319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:54.825567 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.825588 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:05:54.825683 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 07:05:54.825851 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":171,"thread_start_us":108,"threads_started":1}
I20260924 07:05:54.826282 6176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.819274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52536 (local address 127.3.217.2:40169)
0924 07:05:54.820099 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:54.820112 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:54.820140 (+ 28us) server_negotiation.cc:407] Connection header received
0924 07:05:54.820229 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:54.820234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:54.820606 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:54.821171 (+ 565us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:05:54.821306 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.822623 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.824421 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.824970 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.825149 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.825353 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:54.825950 (+ 597us) server_negotiation.cc:299] Negotiation successful
0924 07:05:54.826056 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":176,"threads_started":1}
I20260924 07:05:54.826248 6178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.819947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53284 (local address 127.3.217.3:46651)
0924 07:05:54.820514 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:54.820527 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:54.820554 (+ 27us) server_negotiation.cc:407] Connection header received
0924 07:05:54.820637 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:54.820644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:54.820872 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:54.821023 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:05:54.821233 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.822299 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.824104 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.825012 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.825501 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.825750 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:54.825819 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 07:05:54.826103 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":100,"threads_started":1}
I20260924 07:05:54.827659 6177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.819861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:42453 (local address 127.0.0.1:41156)
0924 07:05:54.821234 (+ 1373us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:54.821250 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:54.821358 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:54.821945 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:54.821948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:54.821967 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:05:54.822059 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:54.822065 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.823936 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.823939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:54.825384 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:54.825396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.827016 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.827020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:54.827200 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.827231 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 07:05:54.827327 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 07:05:54.827426 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1300,"mutex_wait_us":33,"thread_start_us":98,"threads_started":1}
I20260924 07:05:54.827824 6179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.820339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41156 (local address 127.3.217.1:42453)
0924 07:05:54.820815 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:54.820832 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:54.821348 (+ 516us) server_negotiation.cc:407] Connection header received
0924 07:05:54.821477 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:54.821486 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:54.821663 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:54.821975 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:05:54.822173 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.823837 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.825648 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.826542 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.826652 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.826830 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:54.827615 (+ 785us) server_negotiation.cc:299] Negotiation successful
0924 07:05:54.827698 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":169,"threads_started":1}
I20260924 07:05:54.828585 6174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.819059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:40169 (local address 127.0.0.1:52536)
0924 07:05:54.819495 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:54.819596 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:54.819772 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:54.820827 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:54.820833 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:54.820861 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:05:54.821006 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:54.821013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.822899 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.822904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:54.824304 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:54.824312 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.825155 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.825160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:54.825351 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.825558 (+ 207us) client_negotiation.cc:769] Sending connection context
0924 07:05:54.825680 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 07:05:54.825851 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":157,"thread_start_us":67,"threads_started":1}
I20260924 07:05:54.830120 6091 tablet_service.cc:1467] Processing CreateTablet for tablet 9cdf97df3e4d49328d8cc17b3f703c24 (DEFAULT_TABLE table=test-table [id=2f57f543ee384fb78876838155f6a228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:05:54.830606 5811 tablet_service.cc:1467] Processing CreateTablet for tablet 9cdf97df3e4d49328d8cc17b3f703c24 (DEFAULT_TABLE table=test-table [id=2f57f543ee384fb78876838155f6a228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:05:54.831300 6091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cdf97df3e4d49328d8cc17b3f703c24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:54.831367 5933 tablet_service.cc:1467] Processing CreateTablet for tablet 9cdf97df3e4d49328d8cc17b3f703c24 (DEFAULT_TABLE table=test-table [id=2f57f543ee384fb78876838155f6a228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:05:54.832010 5811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cdf97df3e4d49328d8cc17b3f703c24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:54.832394 5933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cdf97df3e4d49328d8cc17b3f703c24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:54.837800 6180 tablet_bootstrap.cc:492] T 9cdf97df3e4d49328d8cc17b3f703c24 P 19459ca02e324f3c9492133944ba2763: Bootstrap starting.
I20260924 07:05:54.839321 6181 tablet_bootstrap.cc:492] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0: Bootstrap starting.
I20260924 07:05:54.839865 6182 tablet_bootstrap.cc:492] T 9cdf97df3e4d49328d8cc17b3f703c24 P 96f31cc614d5499293eafa649dd9eb6c: Bootstrap starting.
I20260924 07:05:54.841323 6180 tablet_bootstrap.cc:654] T 9cdf97df3e4d49328d8cc17b3f703c24 P 19459ca02e324f3c9492133944ba2763: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:54.841770 6181 tablet_bootstrap.cc:654] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:54.841884 6182 tablet_bootstrap.cc:654] T 9cdf97df3e4d49328d8cc17b3f703c24 P 96f31cc614d5499293eafa649dd9eb6c: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:54.842204 6180 log.cc:824] T 9cdf97df3e4d49328d8cc17b3f703c24 P 19459ca02e324f3c9492133944ba2763: Log is configured to *not* fsync() on all Append() calls
I20260924 07:05:54.842518 6181 log.cc:824] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0: Log is configured to *not* fsync() on all Append() calls
I20260924 07:05:54.842579 6182 log.cc:824] T 9cdf97df3e4d49328d8cc17b3f703c24 P 96f31cc614d5499293eafa649dd9eb6c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:05:54.844110 6180 tablet_bootstrap.cc:492] T 9cdf97df3e4d49328d8cc17b3f703c24 P 19459ca02e324f3c9492133944ba2763: No bootstrap required, opened a new log
I20260924 07:05:54.844285 6180 ts_tablet_manager.cc:1397] T 9cdf97df3e4d49328d8cc17b3f703c24 P 19459ca02e324f3c9492133944ba2763: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 07:05:54.844483 6182 tablet_bootstrap.cc:492] T 9cdf97df3e4d49328d8cc17b3f703c24 P 96f31cc614d5499293eafa649dd9eb6c: No bootstrap required, opened a new log
I20260924 07:05:54.844640 6182 ts_tablet_manager.cc:1397] T 9cdf97df3e4d49328d8cc17b3f703c24 P 96f31cc614d5499293eafa649dd9eb6c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 07:05:54.847607 6180 raft_consensus.cc:348] T 9cdf97df3e4d49328d8cc17b3f703c24 P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } } peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } }
I20260924 07:05:54.847586 6182 raft_consensus.cc:348] T 9cdf97df3e4d49328d8cc17b3f703c24 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } } peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } }
I20260924 07:05:54.847798 6180 raft_consensus.cc:374] T 9cdf97df3e4d49328d8cc17b3f703c24 P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:54.847800 6182 raft_consensus.cc:374] T 9cdf97df3e4d49328d8cc17b3f703c24 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:54.847836 6180 raft_consensus.cc:729] T 9cdf97df3e4d49328d8cc17b3f703c24 P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19459ca02e324f3c9492133944ba2763, State: Initialized, Role: FOLLOWER
I20260924 07:05:54.847854 6182 raft_consensus.cc:729] T 9cdf97df3e4d49328d8cc17b3f703c24 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96f31cc614d5499293eafa649dd9eb6c, State: Initialized, Role: FOLLOWER
I20260924 07:05:54.848197 6180 consensus_queue.cc:260] T 9cdf97df3e4d49328d8cc17b3f703c24 P 19459ca02e324f3c9492133944ba2763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } } peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } }
I20260924 07:05:54.848202 6182 consensus_queue.cc:260] T 9cdf97df3e4d49328d8cc17b3f703c24 P 96f31cc614d5499293eafa649dd9eb6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } } peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } }
I20260924 07:05:54.848742 6017 heartbeater.cc:503] Master 127.3.217.62:33557 was elected leader, sending a full tablet report...
I20260924 07:05:54.848740 5877 heartbeater.cc:503] Master 127.3.217.62:33557 was elected leader, sending a full tablet report...
I20260924 07:05:54.849018 6180 ts_tablet_manager.cc:1428] T 9cdf97df3e4d49328d8cc17b3f703c24 P 19459ca02e324f3c9492133944ba2763: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 07:05:54.849802 6182 ts_tablet_manager.cc:1428] T 9cdf97df3e4d49328d8cc17b3f703c24 P 96f31cc614d5499293eafa649dd9eb6c: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260924 07:05:54.850494 6181 tablet_bootstrap.cc:492] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0: No bootstrap required, opened a new log
I20260924 07:05:54.850670 6181 ts_tablet_manager.cc:1397] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260924 07:05:54.854465 6181 raft_consensus.cc:348] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } } peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } }
I20260924 07:05:54.854676 6181 raft_consensus.cc:374] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:54.854734 6181 raft_consensus.cc:729] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f818ff85da4dc497a5a05ab87e02b0, State: Initialized, Role: FOLLOWER
I20260924 07:05:54.855098 6181 consensus_queue.cc:260] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } } peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } }
I20260924 07:05:54.855746 6157 heartbeater.cc:503] Master 127.3.217.62:33557 was elected leader, sending a full tablet report...
I20260924 07:05:54.856031 6181 ts_tablet_manager.cc:1428] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260924 07:05:54.920568 6188 raft_consensus.cc:482] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:05:54.920746 6188 raft_consensus.cc:504] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } } peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } }
I20260924 07:05:54.921903 6188 leader_election.cc:288] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96f31cc614d5499293eafa649dd9eb6c (127.3.217.2:40169), 19459ca02e324f3c9492133944ba2763 (127.3.217.1:42453)
I20260924 07:05:54.927073 6179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.922506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:36873 (local address 127.3.217.1:42453)
0924 07:05:54.922632 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:54.922636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:54.922780 (+ 144us) server_negotiation.cc:407] Connection header received
0924 07:05:54.922905 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:54.922908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:54.922954 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:54.923024 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:05:54.923246 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.923960 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.925673 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.926447 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.926558 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.926622 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:54.926870 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 07:05:54.926951 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:05:54.927073 6176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.922279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:51455 (local address 127.3.217.2:40169)
0924 07:05:54.922410 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:54.922415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:54.922428 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:05:54.922479 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:54.922481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:54.922521 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:54.922583 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:05:54.922911 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.923724 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.925688 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.926454 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.926558 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.926630 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:54.926898 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 07:05:54.926968 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 07:05:54.927107 6050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.922159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:40169 (local address 127.3.217.3:51455)
0924 07:05:54.922279 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:54.922292 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:54.922374 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:54.922685 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:54.922688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:54.922707 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:05:54.922824 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:54.922830 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.923843 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.923846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:54.925468 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:54.925475 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.926593 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.926597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:54.926783 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.926801 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 07:05:54.926882 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 07:05:54.926938 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":66}
I20260924 07:05:54.927106 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.922424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:42453 (local address 127.3.217.3:36873)
0924 07:05:54.922698 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:54.922710 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:54.922819 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:54.923030 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:54.923031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:54.923043 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:05:54.923145 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:54.923151 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.924070 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.924074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:54.925428 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:54.925438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.926588 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.926593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:54.926721 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.926771 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 07:05:54.926856 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 07:05:54.926928 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":31,"thread_start_us":107,"threads_started":1}
I20260924 07:05:54.927692 5971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cdf97df3e4d49328d8cc17b3f703c24" candidate_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96f31cc614d5499293eafa649dd9eb6c" is_pre_election: true
I20260924 07:05:54.927982 5971 raft_consensus.cc:2454] T 9cdf97df3e4d49328d8cc17b3f703c24 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6f818ff85da4dc497a5a05ab87e02b0 in term 0.
I20260924 07:05:54.928122 5830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cdf97df3e4d49328d8cc17b3f703c24" candidate_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19459ca02e324f3c9492133944ba2763" is_pre_election: true
I20260924 07:05:54.928375 5830 raft_consensus.cc:2454] T 9cdf97df3e4d49328d8cc17b3f703c24 P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6f818ff85da4dc497a5a05ab87e02b0 in term 0.
I20260924 07:05:54.928448 6045 leader_election.cc:302] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96f31cc614d5499293eafa649dd9eb6c, c6f818ff85da4dc497a5a05ab87e02b0; no voters:
I20260924 07:05:54.928699 6188 raft_consensus.cc:2781] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:05:54.928777 6188 raft_consensus.cc:482] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:05:54.928824 6188 raft_consensus.cc:3037] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:54.929782 6188 raft_consensus.cc:504] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } } peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } }
I20260924 07:05:54.930053 6188 leader_election.cc:288] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0 [CANDIDATE]: Term 1 election: Requested vote from peers 96f31cc614d5499293eafa649dd9eb6c (127.3.217.2:40169), 19459ca02e324f3c9492133944ba2763 (127.3.217.1:42453)
I20260924 07:05:54.930546 5971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cdf97df3e4d49328d8cc17b3f703c24" candidate_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96f31cc614d5499293eafa649dd9eb6c"
I20260924 07:05:54.930562 5830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cdf97df3e4d49328d8cc17b3f703c24" candidate_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19459ca02e324f3c9492133944ba2763"
I20260924 07:05:54.930691 5971 raft_consensus.cc:3037] T 9cdf97df3e4d49328d8cc17b3f703c24 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:54.930749 5830 raft_consensus.cc:3037] T 9cdf97df3e4d49328d8cc17b3f703c24 P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:54.931692 5971 raft_consensus.cc:2454] T 9cdf97df3e4d49328d8cc17b3f703c24 P 96f31cc614d5499293eafa649dd9eb6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6f818ff85da4dc497a5a05ab87e02b0 in term 1.
I20260924 07:05:54.931955 6045 leader_election.cc:302] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96f31cc614d5499293eafa649dd9eb6c, c6f818ff85da4dc497a5a05ab87e02b0; no voters:
I20260924 07:05:54.932096 6188 raft_consensus.cc:2781] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:05:54.932113 5830 raft_consensus.cc:2454] T 9cdf97df3e4d49328d8cc17b3f703c24 P 19459ca02e324f3c9492133944ba2763 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6f818ff85da4dc497a5a05ab87e02b0 in term 1.
I20260924 07:05:54.932281 6188 raft_consensus.cc:686] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0 [term 1 LEADER]: Becoming Leader. State: Replica: c6f818ff85da4dc497a5a05ab87e02b0, State: Running, Role: LEADER
I20260924 07:05:54.932592 6188 consensus_queue.cc:237] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } } peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } }
I20260924 07:05:54.935340 5673 catalog_manager.cc:5495] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6f818ff85da4dc497a5a05ab87e02b0 (127.3.217.3). New cstate: current_term: 1 leader_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } health_report { overall_health: UNKNOWN } } }
Sep 24 07:05:54 dist-test-slave-2p7j krb5kdc[5629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233553, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:05:54.969794 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.961291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:33557 (local address 127.0.0.1:38088)
0924 07:05:54.961627 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:54.961639 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:54.961750 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:54.961943 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:54.961945 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:54.962298 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:05:54.962403 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:54.962409 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.963231 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.963235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:54.963866 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:54.963876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.964640 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.964644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:54.964766 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.965508 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:05:54.965521 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:05:54.966930 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:05:54.968483 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:54.968490 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:54.968495 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:54.968729 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:54.968961 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:05:54.968963 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:05:54.968964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:05:54.969033 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:05:54.969352 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:05:54.969357 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:05:54.969456 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 07:05:54.969546 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 07:05:54.969661 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":107,"threads_started":1}
I20260924 07:05:54.969841 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:54.961522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38088 (local address 127.3.217.62:33557)
0924 07:05:54.961650 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:54.961653 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:54.961705 (+ 52us) server_negotiation.cc:407] Connection header received
0924 07:05:54.961815 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:54.961817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:54.961853 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:54.961923 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:54.962522 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.963131 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.964103 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:54.964414 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:54.964555 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:54.967078 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:54.967081 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:54.967083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:54.967104 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:54.968377 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:54.968818 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:54.968820 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:54.968821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:54.968892 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:54.969091 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:54.969093 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:54.969094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:54.969259 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:54.969328 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:54.969585 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 07:05:54.969728 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":54}
I20260924 07:05:54.973726 5673 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:38088:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 07:05:54.974323 5673 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:05:54.978964 6091 tablet_service.cc:1467] Processing CreateTablet for tablet f5903b5e499c470aa6aea9da5c66fc32 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3124756a70124c698e209a41e91b767c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:05:54.978927 5811 tablet_service.cc:1467] Processing CreateTablet for tablet f5903b5e499c470aa6aea9da5c66fc32 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3124756a70124c698e209a41e91b767c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:05:54.979214 5933 tablet_service.cc:1467] Processing CreateTablet for tablet f5903b5e499c470aa6aea9da5c66fc32 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3124756a70124c698e209a41e91b767c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:05:54.979632 6091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5903b5e499c470aa6aea9da5c66fc32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:54.979632 5811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5903b5e499c470aa6aea9da5c66fc32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:54.979820 5933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5903b5e499c470aa6aea9da5c66fc32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:54.982139 6182 tablet_bootstrap.cc:492] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c: Bootstrap starting.
I20260924 07:05:54.982293 6180 tablet_bootstrap.cc:492] T f5903b5e499c470aa6aea9da5c66fc32 P 19459ca02e324f3c9492133944ba2763: Bootstrap starting.
I20260924 07:05:54.982632 6181 tablet_bootstrap.cc:492] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0: Bootstrap starting.
I20260924 07:05:54.983088 6182 tablet_bootstrap.cc:654] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:54.983285 6180 tablet_bootstrap.cc:654] T f5903b5e499c470aa6aea9da5c66fc32 P 19459ca02e324f3c9492133944ba2763: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:54.983640 6181 tablet_bootstrap.cc:654] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:54.984709 6180 tablet_bootstrap.cc:492] T f5903b5e499c470aa6aea9da5c66fc32 P 19459ca02e324f3c9492133944ba2763: No bootstrap required, opened a new log
I20260924 07:05:54.984730 6181 tablet_bootstrap.cc:492] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0: No bootstrap required, opened a new log
I20260924 07:05:54.984795 6181 ts_tablet_manager.cc:1397] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:05:54.984795 6180 ts_tablet_manager.cc:1397] T f5903b5e499c470aa6aea9da5c66fc32 P 19459ca02e324f3c9492133944ba2763: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 07:05:54.984923 6182 tablet_bootstrap.cc:492] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c: No bootstrap required, opened a new log
I20260924 07:05:54.985001 6182 ts_tablet_manager.cc:1397] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 07:05:54.985184 6181 raft_consensus.cc:348] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:54.985275 6181 raft_consensus.cc:374] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:54.985237 6180 raft_consensus.cc:348] T f5903b5e499c470aa6aea9da5c66fc32 P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:54.985309 6181 raft_consensus.cc:729] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f818ff85da4dc497a5a05ab87e02b0, State: Initialized, Role: FOLLOWER
I20260924 07:05:54.985334 6180 raft_consensus.cc:374] T f5903b5e499c470aa6aea9da5c66fc32 P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:54.985373 6180 raft_consensus.cc:729] T f5903b5e499c470aa6aea9da5c66fc32 P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19459ca02e324f3c9492133944ba2763, State: Initialized, Role: FOLLOWER
I20260924 07:05:54.985384 6182 raft_consensus.cc:348] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:54.985426 6181 consensus_queue.cc:260] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:54.985477 6182 raft_consensus.cc:374] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:54.985508 6182 raft_consensus.cc:729] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96f31cc614d5499293eafa649dd9eb6c, State: Initialized, Role: FOLLOWER
I20260924 07:05:54.985491 6180 consensus_queue.cc:260] T f5903b5e499c470aa6aea9da5c66fc32 P 19459ca02e324f3c9492133944ba2763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:54.985620 6182 consensus_queue.cc:260] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:54.985673 6181 ts_tablet_manager.cc:1428] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:05:54.985728 6180 ts_tablet_manager.cc:1428] T f5903b5e499c470aa6aea9da5c66fc32 P 19459ca02e324f3c9492133944ba2763: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:05:54.985759 6186 tablet_replica.cc:439] T f5903b5e499c470aa6aea9da5c66fc32 P 19459ca02e324f3c9492133944ba2763: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } } }
I20260924 07:05:54.985885 6182 ts_tablet_manager.cc:1428] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 07:05:54.985890 6186 tablet_replica.cc:442] T f5903b5e499c470aa6aea9da5c66fc32 P 19459ca02e324f3c9492133944ba2763: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:05:54.985843 6188 tablet_replica.cc:439] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } } }
I20260924 07:05:54.985934 6188 tablet_replica.cc:442] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:05:54.986094 6187 tablet_replica.cc:439] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } } }
I20260924 07:05:54.986212 6187 tablet_replica.cc:442] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 07:05:55.010416 6158 tablet.cc:2369] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:05:55.034801 6018 tablet.cc:2369] T 9cdf97df3e4d49328d8cc17b3f703c24 P 96f31cc614d5499293eafa649dd9eb6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:05:55.078881 5878 tablet.cc:2369] T 9cdf97df3e4d49328d8cc17b3f703c24 P 19459ca02e324f3c9492133944ba2763: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:05:55.114498 6187 raft_consensus.cc:482] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:05:55.114637 6187 raft_consensus.cc:504] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:55.115718 6187 leader_election.cc:288] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19459ca02e324f3c9492133944ba2763 (127.3.217.1:42453), c6f818ff85da4dc497a5a05ab87e02b0 (127.3.217.3:46651)
I20260924 07:05:55.120824 6178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:55.116162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:59989 (local address 127.3.217.3:46651)
0924 07:05:55.116278 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:55.116281 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:55.116409 (+ 128us) server_negotiation.cc:407] Connection header received
0924 07:05:55.116494 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:55.116497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:55.116538 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:55.116615 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:05:55.116993 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:55.117771 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:55.119429 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:55.120061 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:55.120168 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:55.120231 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:55.120596 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 07:05:55.120687 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 07:05:55.120824 6179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:55.116070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:38463 (local address 127.3.217.1:42453)
0924 07:05:55.116196 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:55.116200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:55.116326 (+ 126us) server_negotiation.cc:407] Connection header received
0924 07:05:55.116483 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:55.116486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:55.116535 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:55.116615 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:05:55.116993 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:55.117771 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:55.119429 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:55.120066 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:55.120168 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:55.120238 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:55.120596 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 07:05:55.120687 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:05:55.120929 6201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:55.115945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:42453 (local address 127.3.217.2:38463)
0924 07:05:55.116245 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:55.116258 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:55.116370 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:55.116674 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:55.116690 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:55.116736 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:05:55.116885 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:55.116891 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:55.117886 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:55.117890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:55.119295 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:55.119306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:55.120181 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:55.120185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:55.120401 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:55.120443 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 07:05:55.120525 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 07:05:55.120748 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":108,"spinlock_wait_cycles":30848,"thread_start_us":86,"threads_started":1}
I20260924 07:05:55.121300 6188 raft_consensus.cc:482] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:05:55.121335 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:55.115956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:46651 (local address 127.3.217.2:59989)
0924 07:05:55.116340 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:55.116352 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:55.116425 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:55.116690 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:55.116692 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:55.116738 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:05:55.116879 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:55.116886 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:55.117886 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:55.117890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:55.119306 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:55.119313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:55.120195 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:55.120199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:55.120406 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:55.120445 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 07:05:55.120525 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 07:05:55.120748 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":365,"spinlock_wait_cycles":4736,"thread_start_us":74,"threads_started":1}
I20260924 07:05:55.121362 5830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5903b5e499c470aa6aea9da5c66fc32" candidate_uuid: "96f31cc614d5499293eafa649dd9eb6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19459ca02e324f3c9492133944ba2763" is_pre_election: true
I20260924 07:05:55.121397 6188 raft_consensus.cc:504] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:55.121493 5830 raft_consensus.cc:2454] T f5903b5e499c470aa6aea9da5c66fc32 P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96f31cc614d5499293eafa649dd9eb6c in term 0.
I20260924 07:05:55.121847 6188 leader_election.cc:288] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19459ca02e324f3c9492133944ba2763 (127.3.217.1:42453), 96f31cc614d5499293eafa649dd9eb6c (127.3.217.2:40169)
I20260924 07:05:55.122062 6111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5903b5e499c470aa6aea9da5c66fc32" candidate_uuid: "96f31cc614d5499293eafa649dd9eb6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" is_pre_election: true
I20260924 07:05:55.122196 5905 leader_election.cc:302] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19459ca02e324f3c9492133944ba2763, 96f31cc614d5499293eafa649dd9eb6c; no voters:
I20260924 07:05:55.122248 6111 raft_consensus.cc:2454] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96f31cc614d5499293eafa649dd9eb6c in term 0.
I20260924 07:05:55.122269 5830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5903b5e499c470aa6aea9da5c66fc32" candidate_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19459ca02e324f3c9492133944ba2763" is_pre_election: true
I20260924 07:05:55.122363 5830 raft_consensus.cc:2454] T f5903b5e499c470aa6aea9da5c66fc32 P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6f818ff85da4dc497a5a05ab87e02b0 in term 0.
I20260924 07:05:55.122385 5971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5903b5e499c470aa6aea9da5c66fc32" candidate_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96f31cc614d5499293eafa649dd9eb6c" is_pre_election: true
I20260924 07:05:55.122501 5971 raft_consensus.cc:2454] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6f818ff85da4dc497a5a05ab87e02b0 in term 0.
I20260924 07:05:55.122774 6187 raft_consensus.cc:2781] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:05:55.122771 6045 leader_election.cc:302] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96f31cc614d5499293eafa649dd9eb6c, c6f818ff85da4dc497a5a05ab87e02b0; no voters:
I20260924 07:05:55.122843 6187 raft_consensus.cc:482] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:05:55.122892 6187 raft_consensus.cc:3037] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:55.122920 6188 raft_consensus.cc:2781] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:05:55.122972 6188 raft_consensus.cc:482] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:05:55.123008 6188 raft_consensus.cc:3037] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:55.124202 6188 raft_consensus.cc:504] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:55.124202 6187 raft_consensus.cc:504] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:55.124591 6188 leader_election.cc:288] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0 [CANDIDATE]: Term 1 election: Requested vote from peers 19459ca02e324f3c9492133944ba2763 (127.3.217.1:42453), 96f31cc614d5499293eafa649dd9eb6c (127.3.217.2:40169)
I20260924 07:05:55.124603 6187 leader_election.cc:288] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c [CANDIDATE]: Term 1 election: Requested vote from peers 19459ca02e324f3c9492133944ba2763 (127.3.217.1:42453), c6f818ff85da4dc497a5a05ab87e02b0 (127.3.217.3:46651)
I20260924 07:05:55.124886 5830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5903b5e499c470aa6aea9da5c66fc32" candidate_uuid: "96f31cc614d5499293eafa649dd9eb6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19459ca02e324f3c9492133944ba2763"
I20260924 07:05:55.124897 5971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5903b5e499c470aa6aea9da5c66fc32" candidate_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96f31cc614d5499293eafa649dd9eb6c"
I20260924 07:05:55.124930 5831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5903b5e499c470aa6aea9da5c66fc32" candidate_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19459ca02e324f3c9492133944ba2763"
I20260924 07:05:55.124951 6111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5903b5e499c470aa6aea9da5c66fc32" candidate_uuid: "96f31cc614d5499293eafa649dd9eb6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6f818ff85da4dc497a5a05ab87e02b0"
I20260924 07:05:55.124993 5830 raft_consensus.cc:3037] T f5903b5e499c470aa6aea9da5c66fc32 P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:55.125051 5971 raft_consensus.cc:2379] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c6f818ff85da4dc497a5a05ab87e02b0 in current term 1: Already voted for candidate 96f31cc614d5499293eafa649dd9eb6c in this term.
I20260924 07:05:55.125077 6111 raft_consensus.cc:2379] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 96f31cc614d5499293eafa649dd9eb6c in current term 1: Already voted for candidate c6f818ff85da4dc497a5a05ab87e02b0 in this term.
I20260924 07:05:55.125591 6045 leader_election.cc:302] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6f818ff85da4dc497a5a05ab87e02b0; no voters: 19459ca02e324f3c9492133944ba2763, 96f31cc614d5499293eafa649dd9eb6c
I20260924 07:05:55.125804 6188 raft_consensus.cc:2726] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 07:05:55.126595 5830 raft_consensus.cc:2454] T f5903b5e499c470aa6aea9da5c66fc32 P 19459ca02e324f3c9492133944ba2763 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96f31cc614d5499293eafa649dd9eb6c in term 1.
I20260924 07:05:55.126822 5905 leader_election.cc:302] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19459ca02e324f3c9492133944ba2763, 96f31cc614d5499293eafa649dd9eb6c; no voters: c6f818ff85da4dc497a5a05ab87e02b0
I20260924 07:05:55.126974 6187 raft_consensus.cc:2781] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:05:55.127130 6187 raft_consensus.cc:686] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c [term 1 LEADER]: Becoming Leader. State: Replica: 96f31cc614d5499293eafa649dd9eb6c, State: Running, Role: LEADER
I20260924 07:05:55.127471 6187 consensus_queue.cc:237] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:55.129458 6203 tablet_replica.cc:439] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c: TxnStatusTablet state changed. Reason: New leader 96f31cc614d5499293eafa649dd9eb6c. Latest consensus state: current_term: 1 leader_uuid: "96f31cc614d5499293eafa649dd9eb6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } } }
I20260924 07:05:55.129559 6203 tablet_replica.cc:442] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:05:55.129810 6205 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 07:05:55.130005 6205 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 07:05:55.130457 5673 catalog_manager.cc:5495] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 96f31cc614d5499293eafa649dd9eb6c (127.3.217.2). New cstate: current_term: 1 leader_uuid: "96f31cc614d5499293eafa649dd9eb6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } health_report { overall_health: UNKNOWN } } }
I20260924 07:05:55.147850 6176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:55.143346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52544 (local address 127.3.217.2:40169)
0924 07:05:55.143569 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:55.143573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:55.143652 (+ 79us) server_negotiation.cc:407] Connection header received
0924 07:05:55.143869 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:55.143871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:55.143938 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:55.144036 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:05:55.144286 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:55.144823 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:55.146053 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:55.146383 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:55.146490 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:55.146881 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:05:55.147428 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:05:55.147562 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:55.147683 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 07:05:55.147741 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 07:05:55.147907 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:55.143235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:40169 (local address 127.0.0.1:52544)
0924 07:05:55.143578 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:55.143596 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:55.143720 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:55.144068 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:55.144071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:55.144079 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:05:55.144191 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:55.144197 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:55.144919 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:55.144922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:55.145914 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:55.145924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:55.146510 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:55.146514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:55.146607 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:55.146661 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:05:55.147551 (+ 890us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:05:55.147565 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 07:05:55.147653 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 07:05:55.147746 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260924 07:05:55.153307 6111 raft_consensus.cc:1261] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0 [term 1 FOLLOWER]: Refusing update from remote peer 96f31cc614d5499293eafa649dd9eb6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:05:55.153307 5830 raft_consensus.cc:1261] T f5903b5e499c470aa6aea9da5c66fc32 P 19459ca02e324f3c9492133944ba2763 [term 1 FOLLOWER]: Refusing update from remote peer 96f31cc614d5499293eafa649dd9eb6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:05:55.154008 6187 consensus_queue.cc:1035] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:05:55.154238 6203 consensus_queue.cc:1035] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:05:55.157811 6188 tablet_replica.cc:439] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0: TxnStatusTablet state changed. Reason: New leader 96f31cc614d5499293eafa649dd9eb6c. Latest consensus state: current_term: 1 leader_uuid: "96f31cc614d5499293eafa649dd9eb6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } } }
I20260924 07:05:55.157964 6188 tablet_replica.cc:442] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:05:55.158154 6186 tablet_replica.cc:439] T f5903b5e499c470aa6aea9da5c66fc32 P 19459ca02e324f3c9492133944ba2763: TxnStatusTablet state changed. Reason: New leader 96f31cc614d5499293eafa649dd9eb6c. Latest consensus state: current_term: 1 leader_uuid: "96f31cc614d5499293eafa649dd9eb6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } } }
I20260924 07:05:55.158264 6186 tablet_replica.cc:442] T f5903b5e499c470aa6aea9da5c66fc32 P 19459ca02e324f3c9492133944ba2763: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:05:55.160900 6208 mvcc.cc:205] Tried to move back new op lower bound from 7332796641898434560 to 7332796641804640256. Current Snapshot: MvccSnapshot[applied={T|T < 7332796641898434560}]
I20260924 07:05:55.161859 6188 tablet_replica.cc:439] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "96f31cc614d5499293eafa649dd9eb6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } } }
I20260924 07:05:55.161973 6188 tablet_replica.cc:442] T f5903b5e499c470aa6aea9da5c66fc32 P c6f818ff85da4dc497a5a05ab87e02b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:05:55.162434 6203 tablet_replica.cc:439] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "96f31cc614d5499293eafa649dd9eb6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } } }
I20260924 07:05:55.162550 6203 tablet_replica.cc:442] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:05:55.162848 6187 tablet_replica.cc:439] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "96f31cc614d5499293eafa649dd9eb6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } } }
I20260924 07:05:55.162940 6187 tablet_replica.cc:442] T f5903b5e499c470aa6aea9da5c66fc32 P 96f31cc614d5499293eafa649dd9eb6c: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:05:55.162964 6186 tablet_replica.cc:439] T f5903b5e499c470aa6aea9da5c66fc32 P 19459ca02e324f3c9492133944ba2763: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "96f31cc614d5499293eafa649dd9eb6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } } }
I20260924 07:05:55.163058 6186 tablet_replica.cc:442] T f5903b5e499c470aa6aea9da5c66fc32 P 19459ca02e324f3c9492133944ba2763: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:05:55.164261 6210 mvcc.cc:205] Tried to move back new op lower bound from 7332796641898434560 to 7332796641804640256. Current Snapshot: MvccSnapshot[applied={T|T < 7332796641898434560}]
I20260924 07:05:55.173316 6178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:55.170059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53294 (local address 127.3.217.3:46651)
0924 07:05:55.170162 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:55.170166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:55.170179 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:05:55.170298 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:55.170301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:55.170348 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:55.170429 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:05:55.170668 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:55.171164 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:55.172027 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:55.172238 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:55.172328 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:55.172576 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:05:55.172943 (+ 367us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:05:55.173035 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:55.173115 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 07:05:55.173166 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 07:05:55.183326 6163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:55.180225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:40169 (local address 127.3.217.3:37663)
0924 07:05:55.180429 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:55.180442 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:55.180532 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:55.180734 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:55.180737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:55.180744 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:05:55.180844 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:55.180850 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:55.181536 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:55.181539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:55.182247 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:55.182256 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:55.182675 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:55.182679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:55.182768 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:55.182808 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:05:55.183085 (+ 277us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:05:55.183092 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 07:05:55.183148 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 07:05:55.183218 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260924 07:05:55.183327 6176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:55.180333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:37663 (local address 127.3.217.2:40169)
0924 07:05:55.180456 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:55.180460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:55.180488 (+ 28us) server_negotiation.cc:407] Connection header received
0924 07:05:55.180614 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:55.180617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:55.180660 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:55.180716 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:05:55.180943 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:55.181449 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:55.182363 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:55.182577 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:55.182668 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:55.182947 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:05:55.183013 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:05:55.183067 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:55.183156 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 07:05:55.183201 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:05:55.190418 5830 raft_consensus.cc:1261] T 9cdf97df3e4d49328d8cc17b3f703c24 P 19459ca02e324f3c9492133944ba2763 [term 1 FOLLOWER]: Refusing update from remote peer c6f818ff85da4dc497a5a05ab87e02b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:05:55.190562 5971 raft_consensus.cc:1261] T 9cdf97df3e4d49328d8cc17b3f703c24 P 96f31cc614d5499293eafa649dd9eb6c [term 1 FOLLOWER]: Refusing update from remote peer c6f818ff85da4dc497a5a05ab87e02b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:05:55.191030 6190 consensus_queue.cc:1035] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:05:55.191206 6188 consensus_queue.cc:1035] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:05:55.196699 6209 mvcc.cc:205] Tried to move back new op lower bound from 7332796642056060928 to 7332796641006354432. Current Snapshot: MvccSnapshot[applied={T|T < 7332796642056060928}]
I20260924 07:05:55.215713 6178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:55.212058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:59905 (local address 127.3.217.3:46651)
0924 07:05:55.212185 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:55.212189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:55.212329 (+ 140us) server_negotiation.cc:407] Connection header received
0924 07:05:55.212423 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:55.212426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:55.212473 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:55.212552 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:05:55.212762 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:55.213572 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:55.214642 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:55.214979 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:55.215078 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:55.215291 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:05:55.215364 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:05:55.215430 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:55.215540 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 07:05:55.215586 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:05:55.215757 6223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:55.211993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:46651 (local address 127.3.217.2:59905)
0924 07:05:55.212257 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:55.212269 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:55.212351 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:55.212554 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:55.212556 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:55.212561 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:05:55.212679 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:55.212685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:55.213690 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:55.213693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:55.214539 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:55.214548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:55.215072 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:55.215075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:55.215140 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:55.215179 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:05:55.215437 (+ 258us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:05:55.215442 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:05:55.215488 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 07:05:55.215578 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":79,"threads_started":1}
I20260924 07:05:55.241241 5673 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:05:55.241451 5673 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:05:55.244380 5673 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9: Sending DeleteTablet for 3 replicas of tablet 9cdf97df3e4d49328d8cc17b3f703c24
I20260924 07:05:55.245406 5811 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cdf97df3e4d49328d8cc17b3f703c24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:05:55 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:41156
I20260924 07:05:55.245555 5933 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cdf97df3e4d49328d8cc17b3f703c24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:05:55 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:52536
I20260924 07:05:55.245949 6227 tablet_replica.cc:330] T 9cdf97df3e4d49328d8cc17b3f703c24 P 19459ca02e324f3c9492133944ba2763: stopping tablet replica
I20260924 07:05:55.246261 6227 raft_consensus.cc:2229] T 9cdf97df3e4d49328d8cc17b3f703c24 P 19459ca02e324f3c9492133944ba2763 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:05:55.246469 6227 raft_consensus.cc:2258] T 9cdf97df3e4d49328d8cc17b3f703c24 P 19459ca02e324f3c9492133944ba2763 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:05:55.246480 6091 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cdf97df3e4d49328d8cc17b3f703c24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:05:55 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:53284
I20260924 07:05:55.247633 5673 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 07:05:55.247910 6227 ts_tablet_manager.cc:1905] T 9cdf97df3e4d49328d8cc17b3f703c24 P 19459ca02e324f3c9492133944ba2763: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 07:05:55.248071 5673 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:05:55.250918 6227 ts_tablet_manager.cc:1918] T 9cdf97df3e4d49328d8cc17b3f703c24 P 19459ca02e324f3c9492133944ba2763: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:05:55.251024 6227 log.cc:1196] T 9cdf97df3e4d49328d8cc17b3f703c24 P 19459ca02e324f3c9492133944ba2763: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-0/wal/wals/9cdf97df3e4d49328d8cc17b3f703c24
I20260924 07:05:55.251304 6227 ts_tablet_manager.cc:1939] T 9cdf97df3e4d49328d8cc17b3f703c24 P 19459ca02e324f3c9492133944ba2763: Deleting consensus metadata
I20260924 07:05:55.252535 5661 catalog_manager.cc:4841] TS 19459ca02e324f3c9492133944ba2763 (127.3.217.1:42453): tablet 9cdf97df3e4d49328d8cc17b3f703c24 (table test-table [id=2f57f543ee384fb78876838155f6a228]) successfully deleted
I20260924 07:05:55.253424 5811 tablet_service.cc:1467] Processing CreateTablet for tablet 5519ee7fa6ca4e09a6f73714864fe9ca (DEFAULT_TABLE table=test-table [id=e0655fb101664fed89e3dd2cf3052723]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:05:55.253435 6091 tablet_service.cc:1467] Processing CreateTablet for tablet 5519ee7fa6ca4e09a6f73714864fe9ca (DEFAULT_TABLE table=test-table [id=e0655fb101664fed89e3dd2cf3052723]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:05:55.253656 5811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5519ee7fa6ca4e09a6f73714864fe9ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:55.253680 6091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5519ee7fa6ca4e09a6f73714864fe9ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:55.255731 6180 tablet_bootstrap.cc:492] T 5519ee7fa6ca4e09a6f73714864fe9ca P 19459ca02e324f3c9492133944ba2763: Bootstrap starting.
I20260924 07:05:55.256206 6181 tablet_bootstrap.cc:492] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0: Bootstrap starting.
I20260924 07:05:55.256878 6180 tablet_bootstrap.cc:654] T 5519ee7fa6ca4e09a6f73714864fe9ca P 19459ca02e324f3c9492133944ba2763: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:55.257261 6181 tablet_bootstrap.cc:654] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:55.258433 5933 tablet_service.cc:1467] Processing CreateTablet for tablet 5519ee7fa6ca4e09a6f73714864fe9ca (DEFAULT_TABLE table=test-table [id=e0655fb101664fed89e3dd2cf3052723]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:05:55.258733 5933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5519ee7fa6ca4e09a6f73714864fe9ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:55.259667 6229 tablet_replica.cc:330] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0: stopping tablet replica
I20260924 07:05:55.259832 6181 tablet_bootstrap.cc:492] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0: No bootstrap required, opened a new log
I20260924 07:05:55.259914 6181 ts_tablet_manager.cc:1397] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 07:05:55.259989 6229 raft_consensus.cc:2229] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:05:55.260329 6229 raft_consensus.cc:2258] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:05:55.260450 6181 raft_consensus.cc:348] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:55.260565 6181 raft_consensus.cc:374] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:55.260597 6181 raft_consensus.cc:729] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f818ff85da4dc497a5a05ab87e02b0, State: Initialized, Role: FOLLOWER
I20260924 07:05:55.260715 6181 consensus_queue.cc:260] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:55.260986 6181 ts_tablet_manager.cc:1428] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 07:05:55.262830 6229 ts_tablet_manager.cc:1905] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:05:55.263605 6182 tablet_bootstrap.cc:492] T 5519ee7fa6ca4e09a6f73714864fe9ca P 96f31cc614d5499293eafa649dd9eb6c: Bootstrap starting.
I20260924 07:05:55.264693 6182 tablet_bootstrap.cc:654] T 5519ee7fa6ca4e09a6f73714864fe9ca P 96f31cc614d5499293eafa649dd9eb6c: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:55.266319 6229 ts_tablet_manager.cc:1918] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:05:55.266422 6229 log.cc:1196] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/9cdf97df3e4d49328d8cc17b3f703c24
I20260924 07:05:55.266434 6182 tablet_bootstrap.cc:492] T 5519ee7fa6ca4e09a6f73714864fe9ca P 96f31cc614d5499293eafa649dd9eb6c: No bootstrap required, opened a new log
I20260924 07:05:55.266518 6182 ts_tablet_manager.cc:1397] T 5519ee7fa6ca4e09a6f73714864fe9ca P 96f31cc614d5499293eafa649dd9eb6c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:05:55.266711 6229 ts_tablet_manager.cc:1939] T 9cdf97df3e4d49328d8cc17b3f703c24 P c6f818ff85da4dc497a5a05ab87e02b0: Deleting consensus metadata
I20260924 07:05:55.266963 6182 raft_consensus.cc:348] T 5519ee7fa6ca4e09a6f73714864fe9ca P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:55.267091 6182 raft_consensus.cc:374] T 5519ee7fa6ca4e09a6f73714864fe9ca P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:55.267124 6182 raft_consensus.cc:729] T 5519ee7fa6ca4e09a6f73714864fe9ca P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96f31cc614d5499293eafa649dd9eb6c, State: Initialized, Role: FOLLOWER
I20260924 07:05:55.267268 6182 consensus_queue.cc:260] T 5519ee7fa6ca4e09a6f73714864fe9ca P 96f31cc614d5499293eafa649dd9eb6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:55.267582 6182 ts_tablet_manager.cc:1428] T 5519ee7fa6ca4e09a6f73714864fe9ca P 96f31cc614d5499293eafa649dd9eb6c: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260924 07:05:55.268494 5660 catalog_manager.cc:4841] TS c6f818ff85da4dc497a5a05ab87e02b0 (127.3.217.3:46651): tablet 9cdf97df3e4d49328d8cc17b3f703c24 (table test-table [id=2f57f543ee384fb78876838155f6a228]) successfully deleted
I20260924 07:05:55.269940 6180 tablet_bootstrap.cc:492] T 5519ee7fa6ca4e09a6f73714864fe9ca P 19459ca02e324f3c9492133944ba2763: No bootstrap required, opened a new log
I20260924 07:05:55.270030 6180 ts_tablet_manager.cc:1397] T 5519ee7fa6ca4e09a6f73714864fe9ca P 19459ca02e324f3c9492133944ba2763: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.001s
I20260924 07:05:55.270438 6180 raft_consensus.cc:348] T 5519ee7fa6ca4e09a6f73714864fe9ca P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:55.270545 6180 raft_consensus.cc:374] T 5519ee7fa6ca4e09a6f73714864fe9ca P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:55.270576 6180 raft_consensus.cc:729] T 5519ee7fa6ca4e09a6f73714864fe9ca P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19459ca02e324f3c9492133944ba2763, State: Initialized, Role: FOLLOWER
I20260924 07:05:55.270716 6180 consensus_queue.cc:260] T 5519ee7fa6ca4e09a6f73714864fe9ca P 19459ca02e324f3c9492133944ba2763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:55.271001 6180 ts_tablet_manager.cc:1428] T 5519ee7fa6ca4e09a6f73714864fe9ca P 19459ca02e324f3c9492133944ba2763: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:05:55.271740 6228 tablet_replica.cc:330] T 9cdf97df3e4d49328d8cc17b3f703c24 P 96f31cc614d5499293eafa649dd9eb6c: stopping tablet replica
I20260924 07:05:55.272038 6228 raft_consensus.cc:2229] T 9cdf97df3e4d49328d8cc17b3f703c24 P 96f31cc614d5499293eafa649dd9eb6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:05:55.272181 6228 raft_consensus.cc:2258] T 9cdf97df3e4d49328d8cc17b3f703c24 P 96f31cc614d5499293eafa649dd9eb6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:05:55.273577 6228 ts_tablet_manager.cc:1905] T 9cdf97df3e4d49328d8cc17b3f703c24 P 96f31cc614d5499293eafa649dd9eb6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:05:55.276749 6228 ts_tablet_manager.cc:1918] T 9cdf97df3e4d49328d8cc17b3f703c24 P 96f31cc614d5499293eafa649dd9eb6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:05:55.276841 6228 log.cc:1196] T 9cdf97df3e4d49328d8cc17b3f703c24 P 96f31cc614d5499293eafa649dd9eb6c: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/9cdf97df3e4d49328d8cc17b3f703c24
I20260924 07:05:55.277115 6228 ts_tablet_manager.cc:1939] T 9cdf97df3e4d49328d8cc17b3f703c24 P 96f31cc614d5499293eafa649dd9eb6c: Deleting consensus metadata
I20260924 07:05:55.278401 5661 catalog_manager.cc:4841] TS 96f31cc614d5499293eafa649dd9eb6c (127.3.217.2:40169): tablet 9cdf97df3e4d49328d8cc17b3f703c24 (table test-table [id=2f57f543ee384fb78876838155f6a228]) successfully deleted
I20260924 07:05:55.394850 6190 raft_consensus.cc:482] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:05:55.394984 6190 raft_consensus.cc:504] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:55.395344 6190 leader_election.cc:288] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19459ca02e324f3c9492133944ba2763 (127.3.217.1:42453), 96f31cc614d5499293eafa649dd9eb6c (127.3.217.2:40169)
I20260924 07:05:55.395787 5830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5519ee7fa6ca4e09a6f73714864fe9ca" candidate_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19459ca02e324f3c9492133944ba2763" is_pre_election: true
I20260924 07:05:55.395854 5971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5519ee7fa6ca4e09a6f73714864fe9ca" candidate_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96f31cc614d5499293eafa649dd9eb6c" is_pre_election: true
I20260924 07:05:55.395936 5830 raft_consensus.cc:2454] T 5519ee7fa6ca4e09a6f73714864fe9ca P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6f818ff85da4dc497a5a05ab87e02b0 in term 0.
I20260924 07:05:55.395972 5971 raft_consensus.cc:2454] T 5519ee7fa6ca4e09a6f73714864fe9ca P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6f818ff85da4dc497a5a05ab87e02b0 in term 0.
I20260924 07:05:55.396220 6045 leader_election.cc:302] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19459ca02e324f3c9492133944ba2763, c6f818ff85da4dc497a5a05ab87e02b0; no voters:
I20260924 07:05:55.396365 6190 raft_consensus.cc:2781] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:05:55.396421 6190 raft_consensus.cc:482] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:05:55.396468 6190 raft_consensus.cc:3037] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:55.397370 6190 raft_consensus.cc:504] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:55.397650 6190 leader_election.cc:288] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0 [CANDIDATE]: Term 1 election: Requested vote from peers 19459ca02e324f3c9492133944ba2763 (127.3.217.1:42453), 96f31cc614d5499293eafa649dd9eb6c (127.3.217.2:40169)
I20260924 07:05:55.397984 5830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5519ee7fa6ca4e09a6f73714864fe9ca" candidate_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19459ca02e324f3c9492133944ba2763"
I20260924 07:05:55.397984 5971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5519ee7fa6ca4e09a6f73714864fe9ca" candidate_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96f31cc614d5499293eafa649dd9eb6c"
I20260924 07:05:55.398082 5830 raft_consensus.cc:3037] T 5519ee7fa6ca4e09a6f73714864fe9ca P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:55.398104 5971 raft_consensus.cc:3037] T 5519ee7fa6ca4e09a6f73714864fe9ca P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:55.398855 5830 raft_consensus.cc:2454] T 5519ee7fa6ca4e09a6f73714864fe9ca P 19459ca02e324f3c9492133944ba2763 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6f818ff85da4dc497a5a05ab87e02b0 in term 1.
I20260924 07:05:55.399114 6045 leader_election.cc:302] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19459ca02e324f3c9492133944ba2763, c6f818ff85da4dc497a5a05ab87e02b0; no voters:
I20260924 07:05:55.399158 5971 raft_consensus.cc:2454] T 5519ee7fa6ca4e09a6f73714864fe9ca P 96f31cc614d5499293eafa649dd9eb6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6f818ff85da4dc497a5a05ab87e02b0 in term 1.
I20260924 07:05:55.399261 6190 raft_consensus.cc:2781] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:05:55.399338 6190 raft_consensus.cc:686] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0 [term 1 LEADER]: Becoming Leader. State: Replica: c6f818ff85da4dc497a5a05ab87e02b0, State: Running, Role: LEADER
I20260924 07:05:55.399466 6190 consensus_queue.cc:237] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:55.401170 5673 catalog_manager.cc:5495] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6f818ff85da4dc497a5a05ab87e02b0 (127.3.217.3). New cstate: current_term: 1 leader_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } health_report { overall_health: HEALTHY } } }
I20260924 07:05:55.437054 5971 raft_consensus.cc:1261] T 5519ee7fa6ca4e09a6f73714864fe9ca P 96f31cc614d5499293eafa649dd9eb6c [term 1 FOLLOWER]: Refusing update from remote peer c6f818ff85da4dc497a5a05ab87e02b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:05:55.437141 5830 raft_consensus.cc:1261] T 5519ee7fa6ca4e09a6f73714864fe9ca P 19459ca02e324f3c9492133944ba2763 [term 1 FOLLOWER]: Refusing update from remote peer c6f818ff85da4dc497a5a05ab87e02b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:05:55.437469 6190 consensus_queue.cc:1035] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:05:55.437665 6224 consensus_queue.cc:1035] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 07:05:55.448861 6210 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 07:05:57.001700 5673 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:05:57.001870 5673 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:05:57.004612 5673 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9: Sending DeleteTablet for 3 replicas of tablet 5519ee7fa6ca4e09a6f73714864fe9ca
I20260924 07:05:57.005162 6091 tablet_service.cc:1514] Processing DeleteTablet for tablet 5519ee7fa6ca4e09a6f73714864fe9ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:05:56 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:53284
I20260924 07:05:57.005308 5933 tablet_service.cc:1514] Processing DeleteTablet for tablet 5519ee7fa6ca4e09a6f73714864fe9ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:05:56 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:52536
I20260924 07:05:57.005342 5811 tablet_service.cc:1514] Processing DeleteTablet for tablet 5519ee7fa6ca4e09a6f73714864fe9ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:05:56 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:41156
I20260924 07:05:57.005522 6257 tablet_replica.cc:330] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0: stopping tablet replica
I20260924 07:05:57.005621 6257 raft_consensus.cc:2229] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:05:57.005805 6257 raft_consensus.cc:2258] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:05:57.005903 6259 tablet_replica.cc:330] T 5519ee7fa6ca4e09a6f73714864fe9ca P 19459ca02e324f3c9492133944ba2763: stopping tablet replica
I20260924 07:05:57.006029 6259 raft_consensus.cc:2229] T 5519ee7fa6ca4e09a6f73714864fe9ca P 19459ca02e324f3c9492133944ba2763 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:05:57.006055 6258 tablet_replica.cc:330] T 5519ee7fa6ca4e09a6f73714864fe9ca P 96f31cc614d5499293eafa649dd9eb6c: stopping tablet replica
I20260924 07:05:57.006139 6259 raft_consensus.cc:2258] T 5519ee7fa6ca4e09a6f73714864fe9ca P 19459ca02e324f3c9492133944ba2763 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:05:57.006309 6258 raft_consensus.cc:2229] T 5519ee7fa6ca4e09a6f73714864fe9ca P 96f31cc614d5499293eafa649dd9eb6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:05:57.006484 6258 raft_consensus.cc:2258] T 5519ee7fa6ca4e09a6f73714864fe9ca P 96f31cc614d5499293eafa649dd9eb6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:05:57.006681 6259 ts_tablet_manager.cc:1905] T 5519ee7fa6ca4e09a6f73714864fe9ca P 19459ca02e324f3c9492133944ba2763: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:05:57.007357 6258 ts_tablet_manager.cc:1905] T 5519ee7fa6ca4e09a6f73714864fe9ca P 96f31cc614d5499293eafa649dd9eb6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:05:57.009230 6259 ts_tablet_manager.cc:1918] T 5519ee7fa6ca4e09a6f73714864fe9ca P 19459ca02e324f3c9492133944ba2763: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 07:05:57.009320 6259 log.cc:1196] T 5519ee7fa6ca4e09a6f73714864fe9ca P 19459ca02e324f3c9492133944ba2763: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-0/wal/wals/5519ee7fa6ca4e09a6f73714864fe9ca
I20260924 07:05:57.009583 6259 ts_tablet_manager.cc:1939] T 5519ee7fa6ca4e09a6f73714864fe9ca P 19459ca02e324f3c9492133944ba2763: Deleting consensus metadata
I20260924 07:05:57.009766 6257 ts_tablet_manager.cc:1905] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:05:57.010432 5661 catalog_manager.cc:4841] TS 19459ca02e324f3c9492133944ba2763 (127.3.217.1:42453): tablet 5519ee7fa6ca4e09a6f73714864fe9ca (table test-table [id=e0655fb101664fed89e3dd2cf3052723]) successfully deleted
I20260924 07:05:57.011266 6258 ts_tablet_manager.cc:1918] T 5519ee7fa6ca4e09a6f73714864fe9ca P 96f31cc614d5499293eafa649dd9eb6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 07:05:57.011349 6258 log.cc:1196] T 5519ee7fa6ca4e09a6f73714864fe9ca P 96f31cc614d5499293eafa649dd9eb6c: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/5519ee7fa6ca4e09a6f73714864fe9ca
I20260924 07:05:57.011628 6258 ts_tablet_manager.cc:1939] T 5519ee7fa6ca4e09a6f73714864fe9ca P 96f31cc614d5499293eafa649dd9eb6c: Deleting consensus metadata
I20260924 07:05:57.012213 6257 ts_tablet_manager.cc:1918] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 07:05:57.012287 6257 log.cc:1196] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/5519ee7fa6ca4e09a6f73714864fe9ca
I20260924 07:05:57.012537 6257 ts_tablet_manager.cc:1939] T 5519ee7fa6ca4e09a6f73714864fe9ca P c6f818ff85da4dc497a5a05ab87e02b0: Deleting consensus metadata
I20260924 07:05:57.012854 5661 catalog_manager.cc:4841] TS 96f31cc614d5499293eafa649dd9eb6c (127.3.217.2:40169): tablet 5519ee7fa6ca4e09a6f73714864fe9ca (table test-table [id=e0655fb101664fed89e3dd2cf3052723]) successfully deleted
I20260924 07:05:57.013212 5660 catalog_manager.cc:4841] TS c6f818ff85da4dc497a5a05ab87e02b0 (127.3.217.3:46651): tablet 5519ee7fa6ca4e09a6f73714864fe9ca (table test-table [id=e0655fb101664fed89e3dd2cf3052723]) successfully deleted
Sep 24 07:05:57 dist-test-slave-2p7j krb5kdc[5629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233557, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 07:05:57 dist-test-slave-2p7j krb5kdc[5629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233557, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:05:57.042984 6266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:57.026508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47982 (local address 127.3.217.62:33557)
0924 07:05:57.026757 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:57.026790 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:57.027107 (+ 317us) server_negotiation.cc:407] Connection header received
0924 07:05:57.027221 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:57.027223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:57.027273 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:57.027375 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:05:57.028414 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:57.028989 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:57.029989 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:57.030251 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:57.030349 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:57.039186 (+ 8837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:05:57.039190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:05:57.039193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:05:57.039230 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:05:57.041099 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:57.041561 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:57.041564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:57.041565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:57.041658 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:05:57.041972 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:05:57.041975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:05:57.041976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:05:57.042191 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:05:57.042306 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:57.042629 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 07:05:57.042781 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":97,"threads_started":1}
I20260924 07:05:57.046185 5673 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:05:57.046809 5673 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:05:57.053223 6091 tablet_service.cc:1467] Processing CreateTablet for tablet bf718ff364fb473290c2b388d4f16a11 (DEFAULT_TABLE table=test-table [id=68d5624a8aa44bd99d37adc0407a6c03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:05:57.053207 5811 tablet_service.cc:1467] Processing CreateTablet for tablet bf718ff364fb473290c2b388d4f16a11 (DEFAULT_TABLE table=test-table [id=68d5624a8aa44bd99d37adc0407a6c03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:05:57.053535 6091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf718ff364fb473290c2b388d4f16a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:57.053602 5811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf718ff364fb473290c2b388d4f16a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:57.055701 6268 tablet_bootstrap.cc:492] T bf718ff364fb473290c2b388d4f16a11 P c6f818ff85da4dc497a5a05ab87e02b0: Bootstrap starting.
I20260924 07:05:57.055847 5933 tablet_service.cc:1467] Processing CreateTablet for tablet bf718ff364fb473290c2b388d4f16a11 (DEFAULT_TABLE table=test-table [id=68d5624a8aa44bd99d37adc0407a6c03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:05:57.056170 5933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf718ff364fb473290c2b388d4f16a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:57.056718 6268 tablet_bootstrap.cc:654] T bf718ff364fb473290c2b388d4f16a11 P c6f818ff85da4dc497a5a05ab87e02b0: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:57.058039 6268 tablet_bootstrap.cc:492] T bf718ff364fb473290c2b388d4f16a11 P c6f818ff85da4dc497a5a05ab87e02b0: No bootstrap required, opened a new log
I20260924 07:05:57.058145 6268 ts_tablet_manager.cc:1397] T bf718ff364fb473290c2b388d4f16a11 P c6f818ff85da4dc497a5a05ab87e02b0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:05:57.058350 6270 tablet_bootstrap.cc:492] T bf718ff364fb473290c2b388d4f16a11 P 19459ca02e324f3c9492133944ba2763: Bootstrap starting.
I20260924 07:05:57.058652 6268 raft_consensus.cc:348] T bf718ff364fb473290c2b388d4f16a11 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:57.058787 6268 raft_consensus.cc:374] T bf718ff364fb473290c2b388d4f16a11 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:57.058818 6268 raft_consensus.cc:729] T bf718ff364fb473290c2b388d4f16a11 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f818ff85da4dc497a5a05ab87e02b0, State: Initialized, Role: FOLLOWER
I20260924 07:05:57.058948 6268 consensus_queue.cc:260] T bf718ff364fb473290c2b388d4f16a11 P c6f818ff85da4dc497a5a05ab87e02b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:57.059233 6268 ts_tablet_manager.cc:1428] T bf718ff364fb473290c2b388d4f16a11 P c6f818ff85da4dc497a5a05ab87e02b0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:05:57.059355 6270 tablet_bootstrap.cc:654] T bf718ff364fb473290c2b388d4f16a11 P 19459ca02e324f3c9492133944ba2763: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:57.059422 6271 tablet_bootstrap.cc:492] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c: Bootstrap starting.
I20260924 07:05:57.060382 6271 tablet_bootstrap.cc:654] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:57.060636 6270 tablet_bootstrap.cc:492] T bf718ff364fb473290c2b388d4f16a11 P 19459ca02e324f3c9492133944ba2763: No bootstrap required, opened a new log
I20260924 07:05:57.060731 6270 ts_tablet_manager.cc:1397] T bf718ff364fb473290c2b388d4f16a11 P 19459ca02e324f3c9492133944ba2763: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:05:57.061158 6270 raft_consensus.cc:348] T bf718ff364fb473290c2b388d4f16a11 P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:57.061291 6270 raft_consensus.cc:374] T bf718ff364fb473290c2b388d4f16a11 P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:57.061334 6270 raft_consensus.cc:729] T bf718ff364fb473290c2b388d4f16a11 P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19459ca02e324f3c9492133944ba2763, State: Initialized, Role: FOLLOWER
I20260924 07:05:57.061460 6270 consensus_queue.cc:260] T bf718ff364fb473290c2b388d4f16a11 P 19459ca02e324f3c9492133944ba2763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:57.061801 6271 tablet_bootstrap.cc:492] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c: No bootstrap required, opened a new log
I20260924 07:05:57.061913 6271 ts_tablet_manager.cc:1397] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:05:57.062211 6270 ts_tablet_manager.cc:1428] T bf718ff364fb473290c2b388d4f16a11 P 19459ca02e324f3c9492133944ba2763: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:05:57.062397 6271 raft_consensus.cc:348] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:57.062546 6271 raft_consensus.cc:374] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:57.062570 6271 raft_consensus.cc:729] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96f31cc614d5499293eafa649dd9eb6c, State: Initialized, Role: FOLLOWER
I20260924 07:05:57.062700 6271 consensus_queue.cc:260] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:57.062965 6271 ts_tablet_manager.cc:1428] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:05:57.130585 6243 raft_consensus.cc:482] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:05:57.130779 6243 raft_consensus.cc:504] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:57.131407 6243 leader_election.cc:288] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19459ca02e324f3c9492133944ba2763 (127.3.217.1:42453), c6f818ff85da4dc497a5a05ab87e02b0 (127.3.217.3:46651)
I20260924 07:05:57.132055 6111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf718ff364fb473290c2b388d4f16a11" candidate_uuid: "96f31cc614d5499293eafa649dd9eb6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" is_pre_election: true
I20260924 07:05:57.132243 6111 raft_consensus.cc:2454] T bf718ff364fb473290c2b388d4f16a11 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96f31cc614d5499293eafa649dd9eb6c in term 0.
I20260924 07:05:57.132236 5831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf718ff364fb473290c2b388d4f16a11" candidate_uuid: "96f31cc614d5499293eafa649dd9eb6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19459ca02e324f3c9492133944ba2763" is_pre_election: true
I20260924 07:05:57.132436 5831 raft_consensus.cc:2454] T bf718ff364fb473290c2b388d4f16a11 P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96f31cc614d5499293eafa649dd9eb6c in term 0.
I20260924 07:05:57.132666 5904 leader_election.cc:302] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96f31cc614d5499293eafa649dd9eb6c, c6f818ff85da4dc497a5a05ab87e02b0; no voters:
I20260924 07:05:57.132889 6243 raft_consensus.cc:2781] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:05:57.132947 6243 raft_consensus.cc:482] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:05:57.132974 6243 raft_consensus.cc:3037] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:57.134217 6243 raft_consensus.cc:504] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:57.134608 6243 leader_election.cc:288] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c [CANDIDATE]: Term 1 election: Requested vote from peers 19459ca02e324f3c9492133944ba2763 (127.3.217.1:42453), c6f818ff85da4dc497a5a05ab87e02b0 (127.3.217.3:46651)
I20260924 07:05:57.134994 6111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf718ff364fb473290c2b388d4f16a11" candidate_uuid: "96f31cc614d5499293eafa649dd9eb6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6f818ff85da4dc497a5a05ab87e02b0"
I20260924 07:05:57.135114 6111 raft_consensus.cc:3037] T bf718ff364fb473290c2b388d4f16a11 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:57.135732 5831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf718ff364fb473290c2b388d4f16a11" candidate_uuid: "96f31cc614d5499293eafa649dd9eb6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19459ca02e324f3c9492133944ba2763"
I20260924 07:05:57.135885 5831 raft_consensus.cc:3037] T bf718ff364fb473290c2b388d4f16a11 P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:57.136309 6111 raft_consensus.cc:2454] T bf718ff364fb473290c2b388d4f16a11 P c6f818ff85da4dc497a5a05ab87e02b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96f31cc614d5499293eafa649dd9eb6c in term 1.
I20260924 07:05:57.136597 5904 leader_election.cc:302] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96f31cc614d5499293eafa649dd9eb6c, c6f818ff85da4dc497a5a05ab87e02b0; no voters:
I20260924 07:05:57.136806 6243 raft_consensus.cc:2781] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:05:57.136962 6243 raft_consensus.cc:686] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c [term 1 LEADER]: Becoming Leader. State: Replica: 96f31cc614d5499293eafa649dd9eb6c, State: Running, Role: LEADER
I20260924 07:05:57.137187 5831 raft_consensus.cc:2454] T bf718ff364fb473290c2b388d4f16a11 P 19459ca02e324f3c9492133944ba2763 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96f31cc614d5499293eafa649dd9eb6c in term 1.
I20260924 07:05:57.137148 6243 consensus_queue.cc:237] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:57.139199 5674 catalog_manager.cc:5495] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 96f31cc614d5499293eafa649dd9eb6c (127.3.217.2). New cstate: current_term: 1 leader_uuid: "96f31cc614d5499293eafa649dd9eb6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } health_report { overall_health: UNKNOWN } } }
I20260924 07:05:57.167575 6277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:57.163337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48412 (local address 127.3.217.2:40169)
0924 07:05:57.163860 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:57.163868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:57.163884 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:05:57.163951 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:57.163954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:57.164010 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:57.164112 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:05:57.164551 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:57.165222 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:57.166184 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:57.166436 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:57.166572 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:57.166905 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:05:57.166999 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:05:57.167062 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:57.167290 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 07:05:57.167351 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":224,"threads_started":1}
I20260924 07:05:57.178325 6277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:57.174914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:37941 (local address 127.3.217.2:40169)
0924 07:05:57.175041 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 07:05:57.175045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:05:57.175221 (+ 176us) server_negotiation.cc:407] Connection header received
0924 07:05:57.175299 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:05:57.175302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:05:57.175365 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:05:57.175539 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:05:57.175716 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:57.176273 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:57.177170 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:05:57.177406 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:05:57.177499 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:57.177767 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:05:57.177857 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:05:57.177923 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:05:57.178040 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 07:05:57.178137 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":47}
I20260924 07:05:57.178340 6280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:57.174805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:40169 (local address 127.3.217.2:37941)
0924 07:05:57.175104 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:57.175116 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:05:57.175214 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:05:57.175469 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:05:57.175471 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:05:57.175478 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:05:57.175638 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:57.175642 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:57.176384 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:57.176386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:57.177027 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:05:57.177035 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:05:57.177524 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:05:57.177527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:05:57.177606 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:05:57.177638 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:05:57.177947 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:05:57.177953 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:05:57.178018 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 07:05:57.178118 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":28,"thread_start_us":99,"threads_started":1}
I20260924 07:05:57.184573 5831 raft_consensus.cc:1261] T bf718ff364fb473290c2b388d4f16a11 P 19459ca02e324f3c9492133944ba2763 [term 1 FOLLOWER]: Refusing update from remote peer 96f31cc614d5499293eafa649dd9eb6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:05:57.184587 6111 raft_consensus.cc:1261] T bf718ff364fb473290c2b388d4f16a11 P c6f818ff85da4dc497a5a05ab87e02b0 [term 1 FOLLOWER]: Refusing update from remote peer 96f31cc614d5499293eafa649dd9eb6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:05:57.185062 6187 consensus_queue.cc:1035] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:05:57.185249 6238 consensus_queue.cc:1035] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:05:57.225580 5674 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:05:57.225809 5674 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:05:57.228266 5674 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9: Sending DeleteTablet for 3 replicas of tablet bf718ff364fb473290c2b388d4f16a11
I20260924 07:05:57.228849 5933 tablet_service.cc:1514] Processing DeleteTablet for tablet bf718ff364fb473290c2b388d4f16a11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:05:57 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:52536
I20260924 07:05:57.228938 5811 tablet_service.cc:1514] Processing DeleteTablet for tablet bf718ff364fb473290c2b388d4f16a11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:05:57 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:41156
I20260924 07:05:57.229110 6259 tablet_replica.cc:330] T bf718ff364fb473290c2b388d4f16a11 P 19459ca02e324f3c9492133944ba2763: stopping tablet replica
I20260924 07:05:57.229076 6258 tablet_replica.cc:330] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c: stopping tablet replica
I20260924 07:05:57.229095 6091 tablet_service.cc:1514] Processing DeleteTablet for tablet bf718ff364fb473290c2b388d4f16a11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:05:57 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:53284
I20260924 07:05:57.229219 6259 raft_consensus.cc:2229] T bf718ff364fb473290c2b388d4f16a11 P 19459ca02e324f3c9492133944ba2763 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:05:57.229256 6258 raft_consensus.cc:2229] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:05:57.229329 6259 raft_consensus.cc:2258] T bf718ff364fb473290c2b388d4f16a11 P 19459ca02e324f3c9492133944ba2763 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:05:57.229470 6257 tablet_replica.cc:330] T bf718ff364fb473290c2b388d4f16a11 P c6f818ff85da4dc497a5a05ab87e02b0: stopping tablet replica
I20260924 07:05:57.229579 6257 raft_consensus.cc:2229] T bf718ff364fb473290c2b388d4f16a11 P c6f818ff85da4dc497a5a05ab87e02b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:05:57.229609 6258 raft_consensus.cc:2258] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:05:57.229681 6257 raft_consensus.cc:2258] T bf718ff364fb473290c2b388d4f16a11 P c6f818ff85da4dc497a5a05ab87e02b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:05:57.230041 5674 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:05:57.230491 5674 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:05:57.230571 6258 ts_tablet_manager.cc:1905] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:05:57.230984 6259 ts_tablet_manager.cc:1905] T bf718ff364fb473290c2b388d4f16a11 P 19459ca02e324f3c9492133944ba2763: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:05:57.233729 6259 ts_tablet_manager.cc:1918] T bf718ff364fb473290c2b388d4f16a11 P 19459ca02e324f3c9492133944ba2763: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:05:57.233799 6259 log.cc:1196] T bf718ff364fb473290c2b388d4f16a11 P 19459ca02e324f3c9492133944ba2763: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-0/wal/wals/bf718ff364fb473290c2b388d4f16a11
I20260924 07:05:57.233908 6258 ts_tablet_manager.cc:1918] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:05:57.234004 6258 log.cc:1196] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/bf718ff364fb473290c2b388d4f16a11
I20260924 07:05:57.234066 6259 ts_tablet_manager.cc:1939] T bf718ff364fb473290c2b388d4f16a11 P 19459ca02e324f3c9492133944ba2763: Deleting consensus metadata
I20260924 07:05:57.234319 6258 ts_tablet_manager.cc:1939] T bf718ff364fb473290c2b388d4f16a11 P 96f31cc614d5499293eafa649dd9eb6c: Deleting consensus metadata
I20260924 07:05:57.234905 6257 ts_tablet_manager.cc:1905] T bf718ff364fb473290c2b388d4f16a11 P c6f818ff85da4dc497a5a05ab87e02b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:05:57.235539 6091 tablet_service.cc:1467] Processing CreateTablet for tablet 7e1eba997c3e47cc82b273bcd7a3db20 (DEFAULT_TABLE table=test-table [id=46a67e9205d04bfebb21798d2bd69b8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:05:57.235757 6091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e1eba997c3e47cc82b273bcd7a3db20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:57.237248 6257 ts_tablet_manager.cc:1918] T bf718ff364fb473290c2b388d4f16a11 P c6f818ff85da4dc497a5a05ab87e02b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:05:57.237306 6257 log.cc:1196] T bf718ff364fb473290c2b388d4f16a11 P c6f818ff85da4dc497a5a05ab87e02b0: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/bf718ff364fb473290c2b388d4f16a11
I20260924 07:05:57.237533 6257 ts_tablet_manager.cc:1939] T bf718ff364fb473290c2b388d4f16a11 P c6f818ff85da4dc497a5a05ab87e02b0: Deleting consensus metadata
I20260924 07:05:57.237875 6268 tablet_bootstrap.cc:492] T 7e1eba997c3e47cc82b273bcd7a3db20 P c6f818ff85da4dc497a5a05ab87e02b0: Bootstrap starting.
I20260924 07:05:57.237941 5661 catalog_manager.cc:4841] TS 19459ca02e324f3c9492133944ba2763 (127.3.217.1:42453): tablet bf718ff364fb473290c2b388d4f16a11 (table test-table [id=68d5624a8aa44bd99d37adc0407a6c03]) successfully deleted
I20260924 07:05:57.238060 5661 catalog_manager.cc:4841] TS 96f31cc614d5499293eafa649dd9eb6c (127.3.217.2:40169): tablet bf718ff364fb473290c2b388d4f16a11 (table test-table [id=68d5624a8aa44bd99d37adc0407a6c03]) successfully deleted
I20260924 07:05:57.238492 5811 tablet_service.cc:1467] Processing CreateTablet for tablet 7e1eba997c3e47cc82b273bcd7a3db20 (DEFAULT_TABLE table=test-table [id=46a67e9205d04bfebb21798d2bd69b8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:05:57.238812 5811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e1eba997c3e47cc82b273bcd7a3db20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:57.238968 6268 tablet_bootstrap.cc:654] T 7e1eba997c3e47cc82b273bcd7a3db20 P c6f818ff85da4dc497a5a05ab87e02b0: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:57.240139 5660 catalog_manager.cc:4841] TS c6f818ff85da4dc497a5a05ab87e02b0 (127.3.217.3:46651): tablet bf718ff364fb473290c2b388d4f16a11 (table test-table [id=68d5624a8aa44bd99d37adc0407a6c03]) successfully deleted
I20260924 07:05:57.240455 6268 tablet_bootstrap.cc:492] T 7e1eba997c3e47cc82b273bcd7a3db20 P c6f818ff85da4dc497a5a05ab87e02b0: No bootstrap required, opened a new log
I20260924 07:05:57.240540 6268 ts_tablet_manager.cc:1397] T 7e1eba997c3e47cc82b273bcd7a3db20 P c6f818ff85da4dc497a5a05ab87e02b0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:05:57.240898 6270 tablet_bootstrap.cc:492] T 7e1eba997c3e47cc82b273bcd7a3db20 P 19459ca02e324f3c9492133944ba2763: Bootstrap starting.
I20260924 07:05:57.240957 6268 raft_consensus.cc:348] T 7e1eba997c3e47cc82b273bcd7a3db20 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:57.241077 6268 raft_consensus.cc:374] T 7e1eba997c3e47cc82b273bcd7a3db20 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:57.241109 6268 raft_consensus.cc:729] T 7e1eba997c3e47cc82b273bcd7a3db20 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f818ff85da4dc497a5a05ab87e02b0, State: Initialized, Role: FOLLOWER
I20260924 07:05:57.241238 6268 consensus_queue.cc:260] T 7e1eba997c3e47cc82b273bcd7a3db20 P c6f818ff85da4dc497a5a05ab87e02b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:57.241505 6268 ts_tablet_manager.cc:1428] T 7e1eba997c3e47cc82b273bcd7a3db20 P c6f818ff85da4dc497a5a05ab87e02b0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:05:57.241788 6270 tablet_bootstrap.cc:654] T 7e1eba997c3e47cc82b273bcd7a3db20 P 19459ca02e324f3c9492133944ba2763: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:57.242607 5933 tablet_service.cc:1467] Processing CreateTablet for tablet 7e1eba997c3e47cc82b273bcd7a3db20 (DEFAULT_TABLE table=test-table [id=46a67e9205d04bfebb21798d2bd69b8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:05:57.242868 5933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e1eba997c3e47cc82b273bcd7a3db20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:05:57.245119 6271 tablet_bootstrap.cc:492] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c: Bootstrap starting.
I20260924 07:05:57.245905 6271 tablet_bootstrap.cc:654] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c: Neither blocks nor log segments found. Creating new log.
I20260924 07:05:57.246061 6270 tablet_bootstrap.cc:492] T 7e1eba997c3e47cc82b273bcd7a3db20 P 19459ca02e324f3c9492133944ba2763: No bootstrap required, opened a new log
I20260924 07:05:57.246132 6270 ts_tablet_manager.cc:1397] T 7e1eba997c3e47cc82b273bcd7a3db20 P 19459ca02e324f3c9492133944ba2763: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 07:05:57.246547 6270 raft_consensus.cc:348] T 7e1eba997c3e47cc82b273bcd7a3db20 P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:57.246652 6270 raft_consensus.cc:374] T 7e1eba997c3e47cc82b273bcd7a3db20 P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:57.246686 6270 raft_consensus.cc:729] T 7e1eba997c3e47cc82b273bcd7a3db20 P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19459ca02e324f3c9492133944ba2763, State: Initialized, Role: FOLLOWER
I20260924 07:05:57.246824 6270 consensus_queue.cc:260] T 7e1eba997c3e47cc82b273bcd7a3db20 P 19459ca02e324f3c9492133944ba2763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:57.247066 6270 ts_tablet_manager.cc:1428] T 7e1eba997c3e47cc82b273bcd7a3db20 P 19459ca02e324f3c9492133944ba2763: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:05:57.247292 6271 tablet_bootstrap.cc:492] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c: No bootstrap required, opened a new log
I20260924 07:05:57.247479 6271 ts_tablet_manager.cc:1397] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:05:57.248145 6271 raft_consensus.cc:348] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:57.248267 6271 raft_consensus.cc:374] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:05:57.248301 6271 raft_consensus.cc:729] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96f31cc614d5499293eafa649dd9eb6c, State: Initialized, Role: FOLLOWER
I20260924 07:05:57.248421 6271 consensus_queue.cc:260] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:57.248682 6271 ts_tablet_manager.cc:1428] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:05:57.321342 6243 raft_consensus.cc:482] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:05:57.321485 6243 raft_consensus.cc:504] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:57.322001 6243 leader_election.cc:288] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19459ca02e324f3c9492133944ba2763 (127.3.217.1:42453), c6f818ff85da4dc497a5a05ab87e02b0 (127.3.217.3:46651)
I20260924 07:05:57.322485 5831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e1eba997c3e47cc82b273bcd7a3db20" candidate_uuid: "96f31cc614d5499293eafa649dd9eb6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19459ca02e324f3c9492133944ba2763" is_pre_election: true
I20260924 07:05:57.322640 5831 raft_consensus.cc:2454] T 7e1eba997c3e47cc82b273bcd7a3db20 P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96f31cc614d5499293eafa649dd9eb6c in term 0.
I20260924 07:05:57.322803 6111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e1eba997c3e47cc82b273bcd7a3db20" candidate_uuid: "96f31cc614d5499293eafa649dd9eb6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" is_pre_election: true
I20260924 07:05:57.322938 6111 raft_consensus.cc:2454] T 7e1eba997c3e47cc82b273bcd7a3db20 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96f31cc614d5499293eafa649dd9eb6c in term 0.
I20260924 07:05:57.322963 5905 leader_election.cc:302] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19459ca02e324f3c9492133944ba2763, 96f31cc614d5499293eafa649dd9eb6c; no voters:
I20260924 07:05:57.323135 6243 raft_consensus.cc:2781] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:05:57.323189 6243 raft_consensus.cc:482] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:05:57.323235 6243 raft_consensus.cc:3037] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:57.324311 6243 raft_consensus.cc:504] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:57.324606 6243 leader_election.cc:288] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c [CANDIDATE]: Term 1 election: Requested vote from peers 19459ca02e324f3c9492133944ba2763 (127.3.217.1:42453), c6f818ff85da4dc497a5a05ab87e02b0 (127.3.217.3:46651)
I20260924 07:05:57.324867 5831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e1eba997c3e47cc82b273bcd7a3db20" candidate_uuid: "96f31cc614d5499293eafa649dd9eb6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19459ca02e324f3c9492133944ba2763"
I20260924 07:05:57.324975 5831 raft_consensus.cc:3037] T 7e1eba997c3e47cc82b273bcd7a3db20 P 19459ca02e324f3c9492133944ba2763 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:57.325193 6111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e1eba997c3e47cc82b273bcd7a3db20" candidate_uuid: "96f31cc614d5499293eafa649dd9eb6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6f818ff85da4dc497a5a05ab87e02b0"
I20260924 07:05:57.325382 6111 raft_consensus.cc:3037] T 7e1eba997c3e47cc82b273bcd7a3db20 P c6f818ff85da4dc497a5a05ab87e02b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:05:57.325843 5831 raft_consensus.cc:2454] T 7e1eba997c3e47cc82b273bcd7a3db20 P 19459ca02e324f3c9492133944ba2763 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96f31cc614d5499293eafa649dd9eb6c in term 1.
I20260924 07:05:57.326130 5905 leader_election.cc:302] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19459ca02e324f3c9492133944ba2763, 96f31cc614d5499293eafa649dd9eb6c; no voters:
I20260924 07:05:57.326287 6243 raft_consensus.cc:2781] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:05:57.326355 6243 raft_consensus.cc:686] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c [term 1 LEADER]: Becoming Leader. State: Replica: 96f31cc614d5499293eafa649dd9eb6c, State: Running, Role: LEADER
I20260924 07:05:57.326468 6243 consensus_queue.cc:237] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } }
I20260924 07:05:57.327015 6111 raft_consensus.cc:2454] T 7e1eba997c3e47cc82b273bcd7a3db20 P c6f818ff85da4dc497a5a05ab87e02b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96f31cc614d5499293eafa649dd9eb6c in term 1.
I20260924 07:05:57.328596 5673 catalog_manager.cc:5495] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 96f31cc614d5499293eafa649dd9eb6c (127.3.217.2). New cstate: current_term: 1 leader_uuid: "96f31cc614d5499293eafa649dd9eb6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96f31cc614d5499293eafa649dd9eb6c" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 40169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 } health_report { overall_health: UNKNOWN } } }
I20260924 07:05:57.352694 6110 raft_consensus.cc:1261] T 7e1eba997c3e47cc82b273bcd7a3db20 P c6f818ff85da4dc497a5a05ab87e02b0 [term 1 FOLLOWER]: Refusing update from remote peer 96f31cc614d5499293eafa649dd9eb6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:05:57.352850 5831 raft_consensus.cc:1261] T 7e1eba997c3e47cc82b273bcd7a3db20 P 19459ca02e324f3c9492133944ba2763 [term 1 FOLLOWER]: Refusing update from remote peer 96f31cc614d5499293eafa649dd9eb6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:05:57.353214 6187 consensus_queue.cc:1035] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19459ca02e324f3c9492133944ba2763" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:05:57.353404 6238 consensus_queue.cc:1035] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6f818ff85da4dc497a5a05ab87e02b0" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 46651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 07:05:57.367161 6236 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 07:05:58.917361 5674 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:05:58.917514 5674 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:05:58.919715 5674 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7677ccc4700f44efa07ff885f44f8bc9: Sending DeleteTablet for 3 replicas of tablet 7e1eba997c3e47cc82b273bcd7a3db20
I20260924 07:05:58.920377 5811 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1eba997c3e47cc82b273bcd7a3db20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:05:58 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:41156
I20260924 07:05:58.920572 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 5746
I20260924 07:05:58.920584 6091 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1eba997c3e47cc82b273bcd7a3db20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:05:58 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:53284
I20260924 07:05:58.920668 5933 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e1eba997c3e47cc82b273bcd7a3db20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:05:58 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:52536
I20260924 07:05:58.922117 6313 tablet_replica.cc:330] T 7e1eba997c3e47cc82b273bcd7a3db20 P c6f818ff85da4dc497a5a05ab87e02b0: stopping tablet replica
I20260924 07:05:58.922343 6313 raft_consensus.cc:2229] T 7e1eba997c3e47cc82b273bcd7a3db20 P c6f818ff85da4dc497a5a05ab87e02b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:05:58.922474 6313 raft_consensus.cc:2258] T 7e1eba997c3e47cc82b273bcd7a3db20 P c6f818ff85da4dc497a5a05ab87e02b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:05:58.922710 6314 tablet_replica.cc:330] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c: stopping tablet replica
I20260924 07:05:58.922830 6314 raft_consensus.cc:2229] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:05:58.922994 6314 raft_consensus.cc:2258] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:05:58.923534 6313 ts_tablet_manager.cc:1905] T 7e1eba997c3e47cc82b273bcd7a3db20 P c6f818ff85da4dc497a5a05ab87e02b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:05:58.923578 6314 ts_tablet_manager.cc:1905] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:05:58.926313 6314 ts_tablet_manager.cc:1918] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 07:05:58.926314 6313 ts_tablet_manager.cc:1918] T 7e1eba997c3e47cc82b273bcd7a3db20 P c6f818ff85da4dc497a5a05ab87e02b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 07:05:58.926390 6313 log.cc:1196] T 7e1eba997c3e47cc82b273bcd7a3db20 P c6f818ff85da4dc497a5a05ab87e02b0: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/7e1eba997c3e47cc82b273bcd7a3db20
I20260924 07:05:58.926390 6314 log.cc:1196] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/7e1eba997c3e47cc82b273bcd7a3db20
I20260924 07:05:58.926659 6314 ts_tablet_manager.cc:1939] T 7e1eba997c3e47cc82b273bcd7a3db20 P 96f31cc614d5499293eafa649dd9eb6c: Deleting consensus metadata
I20260924 07:05:58.926659 6313 ts_tablet_manager.cc:1939] T 7e1eba997c3e47cc82b273bcd7a3db20 P c6f818ff85da4dc497a5a05ab87e02b0: Deleting consensus metadata
I20260924 07:05:58.927281 5660 catalog_manager.cc:4841] TS c6f818ff85da4dc497a5a05ab87e02b0 (127.3.217.3:46651): tablet 7e1eba997c3e47cc82b273bcd7a3db20 (table test-table [id=46a67e9205d04bfebb21798d2bd69b8f]) successfully deleted
I20260924 07:05:58.927313 5661 catalog_manager.cc:4841] TS 96f31cc614d5499293eafa649dd9eb6c (127.3.217.2:40169): tablet 7e1eba997c3e47cc82b273bcd7a3db20 (table test-table [id=46a67e9205d04bfebb21798d2bd69b8f]) successfully deleted
W20260924 07:05:58.927639 5661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.217.1:42453 (error 108)
I20260924 07:05:58.927798 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 5886
I20260924 07:05:58.932443 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:05:58.932031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:42453 (local address 127.0.0.1:48078)
0924 07:05:58.932292 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 07:05:58.932361 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.1:42453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":89,"threads_started":1}
W20260924 07:05:58.932587 5661 catalog_manager.cc:4584] TS 19459ca02e324f3c9492133944ba2763 (127.3.217.1:42453): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e1eba997c3e47cc82b273bcd7a3db20: Network error: Client connection negotiation failed: client connection to 127.3.217.1:42453: connect: Connection refused (error 111)
I20260924 07:05:58.934311 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 6026
I20260924 07:05:58.941824 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 5642
2026-09-24T07:05:58Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6169 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:05:59 dist-test-slave-2p7j krb5kdc[6319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:05:59 dist-test-slave-2p7j krb5kdc[6319](info): set up 2 sockets
Sep 24 07:05:59 dist-test-slave-2p7j krb5kdc[6319](info): commencing operation
krb5kdc: starting...
W20260924 07:05:59.904254 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:05:59 dist-test-slave-2p7j krb5kdc[6319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:05:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:05:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:06:00.020653 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:43453
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:43453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:00.124560 6332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:00.124785 6332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:00.124842 6332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:00.128665 6332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:06:00.128712 6332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:00.128736 6332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:00.128757 6332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:06:00.128784 6332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:06:00.133811 6332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:43453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:43453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:00.134716 6332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:00.135772 6332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:00.144775 6337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:00.144784 6338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:00.144784 6340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:00.145023 6332 server_base.cc:1034] running on GCE node
I20260924 07:06:00.145591 6332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:00.146556 6332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:00.147703 6332 hybrid_clock.cc:648] HybridClock initialized: now 1790233560147683 us; error 30 us; skew 500 ppm
Sep 24 07:06:00 dist-test-slave-2p7j krb5kdc[6319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233560, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:00.150306 6332 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:06:00.151109 6332 webserver.cc:460] Webserver started at http://127.3.217.62:40283/ using document root <none> and password file <none>
I20260924 07:06:00.151651 6332 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:00.151738 6332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:00.151926 6332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:00.153746 6332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "80c070ebb2414991840e8dd5594a72f6"
format_stamp: "Formatted at 2026-09-24 07:06:00 on dist-test-slave-2p7j"
server_key: "f88b72ea76b3f2ac7d21328d8e9a5d71"
server_key_iv: "c1ff8088b072aab2f091d97cd112d4f8"
server_key_version: "encryptionkey@0"
I20260924 07:06:00.154232 6332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "80c070ebb2414991840e8dd5594a72f6"
format_stamp: "Formatted at 2026-09-24 07:06:00 on dist-test-slave-2p7j"
server_key: "f88b72ea76b3f2ac7d21328d8e9a5d71"
server_key_iv: "c1ff8088b072aab2f091d97cd112d4f8"
server_key_version: "encryptionkey@0"
I20260924 07:06:00.157554 6332 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 07:06:00.159787 6347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:00.160985 6332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260924 07:06:00.161089 6332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "80c070ebb2414991840e8dd5594a72f6"
format_stamp: "Formatted at 2026-09-24 07:06:00 on dist-test-slave-2p7j"
server_key: "f88b72ea76b3f2ac7d21328d8e9a5d71"
server_key_iv: "c1ff8088b072aab2f091d97cd112d4f8"
server_key_version: "encryptionkey@0"
I20260924 07:06:00.161186 6332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:00.174170 6332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:00.176934 6332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:00.177089 6332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:00.184065 6332 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:43453
I20260924 07:06:00.184134 6399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:43453 every 8 connection(s)
I20260924 07:06:00.184918 6332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:06:00.185498 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 6332
I20260924 07:06:00.185609 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:06:00.185912 3940 external_mini_cluster.cc:1428] Setting key d2a158c05c99d886570b18a7a4b0775b
I20260924 07:06:00.188042 6400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:00.193308 6400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80c070ebb2414991840e8dd5594a72f6: Bootstrap starting.
Sep 24 07:06:00 dist-test-slave-2p7j krb5kdc[6319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:06:00.195653 6400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80c070ebb2414991840e8dd5594a72f6: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:00.196476 6400 log.cc:824] T 00000000000000000000000000000000 P 80c070ebb2414991840e8dd5594a72f6: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:00.198647 6400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80c070ebb2414991840e8dd5594a72f6: No bootstrap required, opened a new log
I20260924 07:06:00.200065 6403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:00.187285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56788 (local address 127.3.217.62:43453)
0924 07:06:00.187703 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:00.187718 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:00.187750 (+ 32us) server_negotiation.cc:407] Connection header received
0924 07:06:00.188414 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:00.188431 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:00.188836 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:00.189156 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:00.189967 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:00.190950 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:00.191821 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:00.192283 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:00.192444 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:00.195031 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:00.195037 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:00.195051 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:00.195080 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:00.197047 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:00.197506 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:00.197512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:00.197517 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:00.197642 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:00.197948 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:00.197951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:00.197953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:00.198400 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:00.198573 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:00.198926 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 07:06:00.199194 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":154,"threads_started":1}
I20260924 07:06:00.202512 6400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80c070ebb2414991840e8dd5594a72f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80c070ebb2414991840e8dd5594a72f6" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 43453 } }
I20260924 07:06:00.202699 6400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80c070ebb2414991840e8dd5594a72f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:00.202744 6400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80c070ebb2414991840e8dd5594a72f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c070ebb2414991840e8dd5594a72f6, State: Initialized, Role: FOLLOWER
I20260924 07:06:00.203064 6400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80c070ebb2414991840e8dd5594a72f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80c070ebb2414991840e8dd5594a72f6" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 43453 } }
I20260924 07:06:00.203162 6400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80c070ebb2414991840e8dd5594a72f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:06:00.203202 6400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80c070ebb2414991840e8dd5594a72f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:06:00.203264 6400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80c070ebb2414991840e8dd5594a72f6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:00.204316 6400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80c070ebb2414991840e8dd5594a72f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80c070ebb2414991840e8dd5594a72f6" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 43453 } }
I20260924 07:06:00.204672 6400 leader_election.cc:302] T 00000000000000000000000000000000 P 80c070ebb2414991840e8dd5594a72f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80c070ebb2414991840e8dd5594a72f6; no voters:
I20260924 07:06:00.204968 6400 leader_election.cc:288] T 00000000000000000000000000000000 P 80c070ebb2414991840e8dd5594a72f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:06:00.205042 6405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80c070ebb2414991840e8dd5594a72f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:00.205255 6405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80c070ebb2414991840e8dd5594a72f6 [term 1 LEADER]: Becoming Leader. State: Replica: 80c070ebb2414991840e8dd5594a72f6, State: Running, Role: LEADER
I20260924 07:06:00.205811 6405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80c070ebb2414991840e8dd5594a72f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80c070ebb2414991840e8dd5594a72f6" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 43453 } }
I20260924 07:06:00.206180 6400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80c070ebb2414991840e8dd5594a72f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:06:00.206947 6405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80c070ebb2414991840e8dd5594a72f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80c070ebb2414991840e8dd5594a72f6. Latest consensus state: current_term: 1 leader_uuid: "80c070ebb2414991840e8dd5594a72f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80c070ebb2414991840e8dd5594a72f6" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 43453 } } }
I20260924 07:06:00.207058 6405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80c070ebb2414991840e8dd5594a72f6 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:00.207557 6406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80c070ebb2414991840e8dd5594a72f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80c070ebb2414991840e8dd5594a72f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80c070ebb2414991840e8dd5594a72f6" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 43453 } } }
I20260924 07:06:00.207652 6406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80c070ebb2414991840e8dd5594a72f6 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:00.208029 6414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:06:00.211534 6414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:06:00.217167 6414 catalog_manager.cc:1347] Generated new cluster ID: 8b2dbc2adad4472989e2c954a99b5a4e
I20260924 07:06:00.217242 6414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:06:00.234144 6414 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:06:00.235278 6414 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:06:00.245693 6414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80c070ebb2414991840e8dd5594a72f6: Generated new TSK 0
I20260924 07:06:00.246446 6414 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
I20260924 07:06:00.279006 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:43453
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:00.384235 6426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:00.384480 6426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:00.384546 6426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:00.388477 6426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:00.388536 6426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:00.388626 6426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:06:00.393677 6426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:43453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:00.394574 6426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:00.395493 6426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:00.404500 6431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:00.404489 6432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:00.404489 6434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:00.404873 6426 server_base.cc:1034] running on GCE node
I20260924 07:06:00.405238 6426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:00.405797 6426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:00.407035 6426 hybrid_clock.cc:648] HybridClock initialized: now 1790233560406987 us; error 71 us; skew 500 ppm
Sep 24 07:06:00 dist-test-slave-2p7j krb5kdc[6319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233560, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:00.409889 6426 init.cc:377] Logged in from keytab as kudu/127.3.217.1@KRBTEST.COM (short username kudu)
I20260924 07:06:00.410647 6426 webserver.cc:460] Webserver started at http://127.3.217.1:40013/ using document root <none> and password file <none>
I20260924 07:06:00.411114 6426 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:00.411166 6426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:00.411312 6426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:00.413190 6426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "597d2f2b5c9f4b1f9b7d5a75aee224e7"
format_stamp: "Formatted at 2026-09-24 07:06:00 on dist-test-slave-2p7j"
server_key: "0c435ab76a4385e0ff1eb2c23c16a5d9"
server_key_iv: "a401d6ef5a3ea0970d8ce23059765576"
server_key_version: "encryptionkey@0"
I20260924 07:06:00.413648 6426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "597d2f2b5c9f4b1f9b7d5a75aee224e7"
format_stamp: "Formatted at 2026-09-24 07:06:00 on dist-test-slave-2p7j"
server_key: "0c435ab76a4385e0ff1eb2c23c16a5d9"
server_key_iv: "a401d6ef5a3ea0970d8ce23059765576"
server_key_version: "encryptionkey@0"
I20260924 07:06:00.416968 6426 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 07:06:00.419199 6441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:00.420681 6426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:06:00.420800 6426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "597d2f2b5c9f4b1f9b7d5a75aee224e7"
format_stamp: "Formatted at 2026-09-24 07:06:00 on dist-test-slave-2p7j"
server_key: "0c435ab76a4385e0ff1eb2c23c16a5d9"
server_key_iv: "a401d6ef5a3ea0970d8ce23059765576"
server_key_version: "encryptionkey@0"
I20260924 07:06:00.420899 6426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:00.436602 6426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:00.439257 6426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:00.439441 6426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:00.439980 6426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:00.440728 6426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:00.440775 6426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:00.440824 6426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:00.440852 6426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:00.448951 6426 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:42503
I20260924 07:06:00.449036 6554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:42503 every 8 connection(s)
I20260924 07:06:00.449738 6426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:06:00.454583 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 6426
I20260924 07:06:00.454666 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:06:00.454941 3940 external_mini_cluster.cc:1428] Setting key 2669709d4069afcad53498e8163c8ff3
Sep 24 07:06:00 dist-test-slave-2p7j krb5kdc[6319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233560, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.217.2@KRBTEST.COM; defaulting to no policy
I20260924 07:06:00.463896 6403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:00.451316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:51721 (local address 127.3.217.62:43453)
0924 07:06:00.451444 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:00.451448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:00.452203 (+ 755us) server_negotiation.cc:407] Connection header received
0924 07:06:00.453467 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:00.453470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:00.453524 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:00.453592 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:00.454630 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:00.455223 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:00.456401 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:00.456635 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:00.456723 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:00.460232 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:00.460235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:00.460238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:00.460278 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:00.461820 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:00.462389 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:00.462392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:00.462393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:00.462469 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:00.462785 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:00.462788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:00.462789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:00.462957 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:00.463038 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:00.463590 (+ 552us) server_negotiation.cc:299] Negotiation successful
0924 07:06:00.463731 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:06:00.464633 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:00.451653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:43453 (local address 127.3.217.1:51721)
0924 07:06:00.452069 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:00.452108 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:00.453273 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:00.453756 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:00.453762 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:00.454070 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:00.454495 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:00.454508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:00.455333 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:00.455337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:00.456283 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:00.456293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:00.456766 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:00.456771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:00.456924 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:00.457656 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:00.457677 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:00.459944 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:00.461964 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:00.461970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:00.461985 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:00.462285 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:00.462560 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:00.462563 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:00.462565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:00.462690 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:00.463053 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:00.463060 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:00.463293 (+ 233us) client_negotiation.cc:769] Sending connection context
0924 07:06:00.463484 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 07:06:00.463821 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":93,"threads_started":1}
I20260924 07:06:00.466001 6555 heartbeater.cc:347] Connected to a master server at 127.3.217.62:43453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.2@KRBTEST.COM" created.
I20260924 07:06:00.466228 6555 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:00.466745 6555 heartbeater.cc:511] Master 127.3.217.62:43453 requested a full tablet report, sending...
I20260924 07:06:00.468407 6364 ts_manager.cc:194] Registered new tserver with Master: 597d2f2b5c9f4b1f9b7d5a75aee224e7 (127.3.217.1:42503)
I20260924 07:06:00.469625 6364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.1@KRBTEST.COM'} at 127.3.217.1:51721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
I20260924 07:06:00.484141 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:43453
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:00.591262 6560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:00.591550 6560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:00.591601 6560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:00.596114 6560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:00.596169 6560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:00.596227 6560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:06:00.601966 6560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:43453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:00.603277 6560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:00.604301 6560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:00.615018 6565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:00.615041 6568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:00.615095 6566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:00.615280 6560 server_base.cc:1034] running on GCE node
I20260924 07:06:00.615662 6560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:00.616243 6560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:00.617419 6560 hybrid_clock.cc:648] HybridClock initialized: now 1790233560617400 us; error 33 us; skew 500 ppm
Sep 24 07:06:00 dist-test-slave-2p7j krb5kdc[6319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233560, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:00.620052 6560 init.cc:377] Logged in from keytab as kudu/127.3.217.2@KRBTEST.COM (short username kudu)
I20260924 07:06:00.621023 6560 webserver.cc:460] Webserver started at http://127.3.217.2:37831/ using document root <none> and password file <none>
I20260924 07:06:00.621619 6560 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:00.621683 6560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:00.621843 6560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:00.623895 6560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "53785978692b4277b00c8c3cdd3630e1"
format_stamp: "Formatted at 2026-09-24 07:06:00 on dist-test-slave-2p7j"
server_key: "723bdec5179e83e0af2d4f481d85b967"
server_key_iv: "dee86cf28b807df2021c70e82672db23"
server_key_version: "encryptionkey@0"
I20260924 07:06:00.624444 6560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "53785978692b4277b00c8c3cdd3630e1"
format_stamp: "Formatted at 2026-09-24 07:06:00 on dist-test-slave-2p7j"
server_key: "723bdec5179e83e0af2d4f481d85b967"
server_key_iv: "dee86cf28b807df2021c70e82672db23"
server_key_version: "encryptionkey@0"
I20260924 07:06:00.627995 6560 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:06:00.630721 6575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:00.632351 6560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:06:00.632471 6560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "53785978692b4277b00c8c3cdd3630e1"
format_stamp: "Formatted at 2026-09-24 07:06:00 on dist-test-slave-2p7j"
server_key: "723bdec5179e83e0af2d4f481d85b967"
server_key_iv: "dee86cf28b807df2021c70e82672db23"
server_key_version: "encryptionkey@0"
I20260924 07:06:00.632573 6560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:00.646647 6560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:00.649508 6560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:00.649663 6560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:00.650219 6560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:00.650956 6560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:00.651001 6560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:00.651036 6560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:00.651060 6560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:00.659713 6560 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:40873
I20260924 07:06:00.659813 6688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:40873 every 8 connection(s)
I20260924 07:06:00.660569 6560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:06:00.670158 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 6560
I20260924 07:06:00.670252 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:06:00.670498 3940 external_mini_cluster.cc:1428] Setting key 5811f4ef3db4a9ca8507656237af934d
Sep 24 07:06:00 dist-test-slave-2p7j krb5kdc[6319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233560, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:06:00.675252 6403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:00.662424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:34639 (local address 127.3.217.62:43453)
0924 07:06:00.662613 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:00.662617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:00.663223 (+ 606us) server_negotiation.cc:407] Connection header received
0924 07:06:00.664793 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:00.664795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:00.664843 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:00.664922 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:00.666538 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:00.667165 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:00.668697 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:00.668933 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:00.669018 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:00.671549 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:00.671553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:00.671555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:00.671576 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:00.673005 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:00.673690 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:00.673695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:00.673697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:00.673786 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:00.674128 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:00.674132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:00.674134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:00.674337 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:00.674433 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:00.674928 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 07:06:00.675052 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 07:06:00.676043 6691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:00.662678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:43453 (local address 127.3.217.2:34639)
0924 07:06:00.663083 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:00.663122 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:00.664436 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:00.665213 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:00.665221 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:00.665574 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:00.666315 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:00.666344 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:00.667315 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:00.667319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:00.668543 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:00.668557 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:00.669125 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:00.669129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:00.669304 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:00.669869 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:00.669885 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:00.671394 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:00.673130 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:00.673134 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:00.673146 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:00.673498 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:00.673878 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:00.673882 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:00.673884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:00.674032 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:00.674441 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:00.674448 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:00.674693 (+ 245us) client_negotiation.cc:769] Sending connection context
0924 07:06:00.674867 (+ 174us) client_negotiation.cc:240] Negotiation successful
0924 07:06:00.675182 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":108,"threads_started":1}
I20260924 07:06:00.677770 6689 heartbeater.cc:347] Connected to a master server at 127.3.217.62:43453
I20260924 07:06:00.677991 6689 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:00.678435 6689 heartbeater.cc:511] Master 127.3.217.62:43453 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.217.3@KRBTEST.COM; defaulting to no policy
I20260924 07:06:00.679576 6364 ts_manager.cc:194] Registered new tserver with Master: 53785978692b4277b00c8c3cdd3630e1 (127.3.217.2:40873)
I20260924 07:06:00.680150 6364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.2@KRBTEST.COM'} at 127.3.217.2:34639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
I20260924 07:06:00.698381 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:43453
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:00.805507 6694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:00.805721 6694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:00.805759 6694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:00.809800 6694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:00.809854 6694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:00.809914 6694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:06:00.814903 6694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:43453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:00.815833 6694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:00.816733 6694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:00.826062 6699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:00.826066 6700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:00.826104 6702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:00.826165 6694 server_base.cc:1034] running on GCE node
I20260924 07:06:00.826696 6694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:00.827270 6694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:00.828440 6694 hybrid_clock.cc:648] HybridClock initialized: now 1790233560828403 us; error 44 us; skew 500 ppm
Sep 24 07:06:00 dist-test-slave-2p7j krb5kdc[6319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233560, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:00.831060 6694 init.cc:377] Logged in from keytab as kudu/127.3.217.3@KRBTEST.COM (short username kudu)
I20260924 07:06:00.831889 6694 webserver.cc:460] Webserver started at http://127.3.217.3:37841/ using document root <none> and password file <none>
I20260924 07:06:00.832374 6694 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:00.832429 6694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:00.832577 6694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:00.834501 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "a5db08c0ec0148ccb3dcde4f2534dee2"
format_stamp: "Formatted at 2026-09-24 07:06:00 on dist-test-slave-2p7j"
server_key: "5adb81a2193626d123072829760a263e"
server_key_iv: "25a6bff6c9339af6420557c4805c0fa2"
server_key_version: "encryptionkey@0"
I20260924 07:06:00.834961 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "a5db08c0ec0148ccb3dcde4f2534dee2"
format_stamp: "Formatted at 2026-09-24 07:06:00 on dist-test-slave-2p7j"
server_key: "5adb81a2193626d123072829760a263e"
server_key_iv: "25a6bff6c9339af6420557c4805c0fa2"
server_key_version: "encryptionkey@0"
I20260924 07:06:00.838415 6694 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 07:06:00.840605 6709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:00.841720 6694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:06:00.841825 6694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "a5db08c0ec0148ccb3dcde4f2534dee2"
format_stamp: "Formatted at 2026-09-24 07:06:00 on dist-test-slave-2p7j"
server_key: "5adb81a2193626d123072829760a263e"
server_key_iv: "25a6bff6c9339af6420557c4805c0fa2"
server_key_version: "encryptionkey@0"
I20260924 07:06:00.841925 6694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:00.854290 6694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:00.856905 6694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:00.857049 6694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:00.857549 6694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:00.858270 6694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:00.858315 6694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:00.858353 6694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:00.858382 6694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:00.867280 6694 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:46433
I20260924 07:06:00.867337 6822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:46433 every 8 connection(s)
I20260924 07:06:00.868062 6694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:06:00.874080 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 6694
I20260924 07:06:00.874166 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:06:00.874404 3940 external_mini_cluster.cc:1428] Setting key 70f1ab88331c0cfb092d02035c200c14
Sep 24 07:06:00 dist-test-slave-2p7j krb5kdc[6319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233560, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:06:00.881225 6403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:00.869672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:55385 (local address 127.3.217.62:43453)
0924 07:06:00.869804 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:00.869809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:00.870560 (+ 751us) server_negotiation.cc:407] Connection header received
0924 07:06:00.871822 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:00.871825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:00.871868 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:00.871949 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:00.872970 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:00.873501 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:00.874678 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:00.874944 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:00.875031 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:00.877565 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:00.877569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:00.877571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:00.877592 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:00.879028 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:00.879723 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:00.879726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:00.879727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:00.879812 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:00.880183 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:00.880186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:00.880187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:00.880365 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:00.880443 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:00.880981 (+ 538us) server_negotiation.cc:299] Negotiation successful
0924 07:06:00.881093 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:06:00.881927 6825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:00.870013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:43453 (local address 127.3.217.3:55385)
0924 07:06:00.870429 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:00.870467 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:00.871615 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:00.872074 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:00.872080 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:00.872390 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:00.872840 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:00.872853 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:00.873621 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:00.873624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:00.874571 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:00.874580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:00.875088 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:00.875093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:00.875274 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:00.875985 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:00.876004 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:00.877410 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:00.879196 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:00.879200 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:00.879210 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:00.879602 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:00.879933 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:00.879936 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:00.879937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:00.880069 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:00.880474 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:00.880479 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:00.880697 (+ 218us) client_negotiation.cc:769] Sending connection context
0924 07:06:00.880849 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 07:06:00.881088 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":102,"threads_started":1}
I20260924 07:06:00.882982 6823 heartbeater.cc:347] Connected to a master server at 127.3.217.62:43453
I20260924 07:06:00.883188 6823 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:00.883677 6823 heartbeater.cc:511] Master 127.3.217.62:43453 requested a full tablet report, sending...
I20260924 07:06:00.884871 6364 ts_manager.cc:194] Registered new tserver with Master: a5db08c0ec0148ccb3dcde4f2534dee2 (127.3.217.3:46433)
I20260924 07:06:00.885617 6364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.3@KRBTEST.COM'} at 127.3.217.3:55385
I20260924 07:06:00.888005 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 07:06:00.900975 6403 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:00.899277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56794 (local address 127.3.217.62:43453)
0924 07:06:00.899384 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:00.899387 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:00.899553 (+ 166us) server_negotiation.cc:407] Connection header received
0924 07:06:00.899784 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:00.899788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:00.899863 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:00.899989 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:00.900797 (+ 808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56794: BlockingRecv error: recv got EOF from 127.0.0.1:56794 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:06:00.902827 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 6426
I20260924 07:06:00.908231 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 6560
I20260924 07:06:00.912725 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 6694
I20260924 07:06:00.917250 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 6332
2026-09-24T07:06:00Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1973 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:06:00 dist-test-slave-2p7j krb5kdc[6835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:06:00 dist-test-slave-2p7j krb5kdc[6835](info): set up 2 sockets
Sep 24 07:06:00 dist-test-slave-2p7j krb5kdc[6835](info): commencing operation
krb5kdc: starting...
W20260924 07:06:01.887960 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:06:01 dist-test-slave-2p7j krb5kdc[6835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:06:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:06:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:06:02.015462 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:45559
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:41549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:45559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:02.129264 6848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:02.129532 6848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:02.129594 6848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:02.134729 6848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:06:02.134810 6848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:02.134835 6848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:02.134860 6848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:06:02.134882 6848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:06:02.141289 6848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:41549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:45559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:45559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:02.142206 6848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:02.143100 6848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:02.151798 6853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:02.151809 6854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:02.151955 6856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:02.151986 6848 server_base.cc:1034] running on GCE node
I20260924 07:06:02.152820 6848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:02.153841 6848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:02.155062 6848 hybrid_clock.cc:648] HybridClock initialized: now 1790233562154972 us; error 92 us; skew 500 ppm
Sep 24 07:06:02 dist-test-slave-2p7j krb5kdc[6835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233562, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:02.163707 6848 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:06:02.164543 6848 webserver.cc:460] Webserver started at http://127.3.217.62:41649/ using document root <none> and password file <none>
I20260924 07:06:02.165055 6848 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:02.165118 6848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:02.165302 6848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:02.167193 6848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "f8f362c8d4ef403eb61f85d86fa95d91"
format_stamp: "Formatted at 2026-09-24 07:06:02 on dist-test-slave-2p7j"
server_key: "f4b1cd74b53e10a41f9bda0c030604a4"
server_key_iv: "723affecc28ffa6824c201f05b74267f"
server_key_version: "encryptionkey@0"
I20260924 07:06:02.167742 6848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "f8f362c8d4ef403eb61f85d86fa95d91"
format_stamp: "Formatted at 2026-09-24 07:06:02 on dist-test-slave-2p7j"
server_key: "f4b1cd74b53e10a41f9bda0c030604a4"
server_key_iv: "723affecc28ffa6824c201f05b74267f"
server_key_version: "encryptionkey@0"
I20260924 07:06:02.171237 6848 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260924 07:06:02.174067 6863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:02.175274 6848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 07:06:02.175407 6848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "f8f362c8d4ef403eb61f85d86fa95d91"
format_stamp: "Formatted at 2026-09-24 07:06:02 on dist-test-slave-2p7j"
server_key: "f4b1cd74b53e10a41f9bda0c030604a4"
server_key_iv: "723affecc28ffa6824c201f05b74267f"
server_key_version: "encryptionkey@0"
I20260924 07:06:02.175495 6848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:02.201469 6848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:02.204902 6848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:02.205118 6848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:02.215570 6848 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:45559
I20260924 07:06:02.215651 6915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:45559 every 8 connection(s)
I20260924 07:06:02.216568 6848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:06:02.219769 6916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:02.222340 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 6848
I20260924 07:06:02.222496 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:06:02.222873 3940 external_mini_cluster.cc:1428] Setting key de9be75e9f143a8e35b1f026292c2e8e
I20260924 07:06:02.225461 6916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8f362c8d4ef403eb61f85d86fa95d91: Bootstrap starting.
I20260924 07:06:02.227794 6916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8f362c8d4ef403eb61f85d86fa95d91: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:02.228490 6916 log.cc:824] T 00000000000000000000000000000000 P f8f362c8d4ef403eb61f85d86fa95d91: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:02.231195 6916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8f362c8d4ef403eb61f85d86fa95d91: No bootstrap required, opened a new log
Sep 24 07:06:02 dist-test-slave-2p7j krb5kdc[6835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:06:02.235107 6916 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8f362c8d4ef403eb61f85d86fa95d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8f362c8d4ef403eb61f85d86fa95d91" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45559 } }
I20260924 07:06:02.235332 6916 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8f362c8d4ef403eb61f85d86fa95d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:02.235419 6916 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8f362c8d4ef403eb61f85d86fa95d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f362c8d4ef403eb61f85d86fa95d91, State: Initialized, Role: FOLLOWER
I20260924 07:06:02.235971 6916 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8f362c8d4ef403eb61f85d86fa95d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8f362c8d4ef403eb61f85d86fa95d91" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45559 } }
I20260924 07:06:02.236228 6916 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8f362c8d4ef403eb61f85d86fa95d91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:06:02.236312 6916 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8f362c8d4ef403eb61f85d86fa95d91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:06:02.236441 6916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8f362c8d4ef403eb61f85d86fa95d91 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:02.237921 6916 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8f362c8d4ef403eb61f85d86fa95d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8f362c8d4ef403eb61f85d86fa95d91" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45559 } }
I20260924 07:06:02.238425 6916 leader_election.cc:302] T 00000000000000000000000000000000 P f8f362c8d4ef403eb61f85d86fa95d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8f362c8d4ef403eb61f85d86fa95d91; no voters:
I20260924 07:06:02.238969 6916 leader_election.cc:288] T 00000000000000000000000000000000 P f8f362c8d4ef403eb61f85d86fa95d91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:06:02.239145 6921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8f362c8d4ef403eb61f85d86fa95d91 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:02.239444 6921 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8f362c8d4ef403eb61f85d86fa95d91 [term 1 LEADER]: Becoming Leader. State: Replica: f8f362c8d4ef403eb61f85d86fa95d91, State: Running, Role: LEADER
I20260924 07:06:02.239897 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:02.224658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35858 (local address 127.3.217.62:45559)
0924 07:06:02.225039 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:02.225054 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:02.225153 (+ 99us) server_negotiation.cc:407] Connection header received
0924 07:06:02.225789 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:02.225804 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:02.226123 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:02.226379 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:02.227249 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:02.228327 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:02.229506 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:02.230114 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:02.230251 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:02.234156 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:02.234192 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:02.234214 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:02.234260 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:02.236287 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:02.236917 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:02.236924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:02.236930 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:02.237175 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:02.237525 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:02.237527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:02.237528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:02.237970 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:02.238180 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:02.238595 (+ 415us) server_negotiation.cc:299] Negotiation successful
0924 07:06:02.238933 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":102,"threads_started":1}
I20260924 07:06:02.239859 6921 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8f362c8d4ef403eb61f85d86fa95d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8f362c8d4ef403eb61f85d86fa95d91" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45559 } }
I20260924 07:06:02.240489 6916 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8f362c8d4ef403eb61f85d86fa95d91 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:06:02.241496 6922 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8f362c8d4ef403eb61f85d86fa95d91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8f362c8d4ef403eb61f85d86fa95d91. Latest consensus state: current_term: 1 leader_uuid: "f8f362c8d4ef403eb61f85d86fa95d91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8f362c8d4ef403eb61f85d86fa95d91" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45559 } } }
I20260924 07:06:02.241695 6923 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8f362c8d4ef403eb61f85d86fa95d91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8f362c8d4ef403eb61f85d86fa95d91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8f362c8d4ef403eb61f85d86fa95d91" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45559 } } }
I20260924 07:06:02.241789 6922 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8f362c8d4ef403eb61f85d86fa95d91 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:02.241870 6923 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8f362c8d4ef403eb61f85d86fa95d91 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:02.242400 6931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:06:02.242942 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:06:02.245615 6931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:06:02.251978 6931 catalog_manager.cc:1347] Generated new cluster ID: 397fb2e88195437fb9c8bce19681d0a8
I20260924 07:06:02.252142 6931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:06:02.259341 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:02.254653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35862 (local address 127.3.217.62:45559)
0924 07:06:02.254772 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:02.254776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:02.254988 (+ 212us) server_negotiation.cc:407] Connection header received
0924 07:06:02.255123 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:02.255126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:02.255171 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:02.255235 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:02.255807 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:02.256347 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:02.257145 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:02.257420 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:02.257512 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:02.258447 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:02.258451 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:02.258454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:06:02.258909 (+ 455us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:02.258938 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:06:02.258953 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:02.259033 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:02.259131 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 07:06:02.259195 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:06:02.261653 6931 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:06:02.262877 6931 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:06:02.274742 6931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8f362c8d4ef403eb61f85d86fa95d91: Generated new TSK 0
I20260924 07:06:02.275470 6931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:06:02.285840 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 6848
2026-09-24T07:06:02Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1367 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:06:02 dist-test-slave-2p7j krb5kdc[6949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:06:02 dist-test-slave-2p7j krb5kdc[6949](info): set up 2 sockets
Sep 24 07:06:02 dist-test-slave-2p7j krb5kdc[6949](info): commencing operation
krb5kdc: starting...
W20260924 07:06:03.256045 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:06:03 dist-test-slave-2p7j krb5kdc[6949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:06:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:06:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:06:03.386781 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:39783
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:39783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:03.497043 6962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:03.497283 6962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:03.497324 6962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:03.501180 6962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:06:03.501235 6962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:03.501256 6962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:03.501274 6962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:06:03.501291 6962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:06:03.508335 6962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:39783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:39783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:03.509722 6962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:03.511168 6962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:03.522192 6968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:03.522151 6967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:03.522240 6970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:03.522775 6962 server_base.cc:1034] running on GCE node
I20260924 07:06:03.523450 6962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:03.524865 6962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:03.526075 6962 hybrid_clock.cc:648] HybridClock initialized: now 1790233563526050 us; error 37 us; skew 500 ppm
Sep 24 07:06:03 dist-test-slave-2p7j krb5kdc[6949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233563, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:03.529153 6962 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:06:03.530133 6962 webserver.cc:460] Webserver started at http://127.3.217.62:45067/ using document root <none> and password file <none>
I20260924 07:06:03.530889 6962 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:03.530982 6962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:03.531251 6962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:03.533926 6962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "6b69d6faac5942b5ac7b8f4edb57de9b"
format_stamp: "Formatted at 2026-09-24 07:06:03 on dist-test-slave-2p7j"
server_key: "b69b9c27383c4d682499d97fe4de9c80"
server_key_iv: "1b60b4f6a8f4383bcccf2193c73e30ee"
server_key_version: "encryptionkey@0"
I20260924 07:06:03.534595 6962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "6b69d6faac5942b5ac7b8f4edb57de9b"
format_stamp: "Formatted at 2026-09-24 07:06:03 on dist-test-slave-2p7j"
server_key: "b69b9c27383c4d682499d97fe4de9c80"
server_key_iv: "1b60b4f6a8f4383bcccf2193c73e30ee"
server_key_version: "encryptionkey@0"
I20260924 07:06:03.539260 6962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 07:06:03.542315 6977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:03.543671 6962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:06:03.543782 6962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "6b69d6faac5942b5ac7b8f4edb57de9b"
format_stamp: "Formatted at 2026-09-24 07:06:03 on dist-test-slave-2p7j"
server_key: "b69b9c27383c4d682499d97fe4de9c80"
server_key_iv: "1b60b4f6a8f4383bcccf2193c73e30ee"
server_key_version: "encryptionkey@0"
I20260924 07:06:03.543905 6962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:03.585590 6962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:03.588609 6962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:03.588788 6962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:03.595994 6962 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:39783
I20260924 07:06:03.596006 7029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:39783 every 8 connection(s)
I20260924 07:06:03.596930 6962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:06:03.599591 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:03.602761 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 6962
I20260924 07:06:03.602875 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:06:03.603137 3940 external_mini_cluster.cc:1428] Setting key 9cb1b60d121667420eb3f355cef4b6aa
I20260924 07:06:03.605219 7030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b69d6faac5942b5ac7b8f4edb57de9b: Bootstrap starting.
I20260924 07:06:03.607496 7030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b69d6faac5942b5ac7b8f4edb57de9b: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:03.608249 7030 log.cc:824] T 00000000000000000000000000000000 P 6b69d6faac5942b5ac7b8f4edb57de9b: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:03.610409 7030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b69d6faac5942b5ac7b8f4edb57de9b: No bootstrap required, opened a new log
Sep 24 07:06:03 dist-test-slave-2p7j krb5kdc[6949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:06:03.614406 7030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b69d6faac5942b5ac7b8f4edb57de9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b69d6faac5942b5ac7b8f4edb57de9b" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 39783 } }
I20260924 07:06:03.614651 7030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b69d6faac5942b5ac7b8f4edb57de9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:03.614704 7030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b69d6faac5942b5ac7b8f4edb57de9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b69d6faac5942b5ac7b8f4edb57de9b, State: Initialized, Role: FOLLOWER
I20260924 07:06:03.615109 7030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b69d6faac5942b5ac7b8f4edb57de9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b69d6faac5942b5ac7b8f4edb57de9b" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 39783 } }
I20260924 07:06:03.615231 7030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b69d6faac5942b5ac7b8f4edb57de9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:06:03.615293 7030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b69d6faac5942b5ac7b8f4edb57de9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:06:03.615384 7030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b69d6faac5942b5ac7b8f4edb57de9b [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:03.616628 7030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b69d6faac5942b5ac7b8f4edb57de9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b69d6faac5942b5ac7b8f4edb57de9b" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 39783 } }
I20260924 07:06:03.617101 7030 leader_election.cc:302] T 00000000000000000000000000000000 P 6b69d6faac5942b5ac7b8f4edb57de9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b69d6faac5942b5ac7b8f4edb57de9b; no voters:
I20260924 07:06:03.617493 7030 leader_election.cc:288] T 00000000000000000000000000000000 P 6b69d6faac5942b5ac7b8f4edb57de9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:06:03.617671 7035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b69d6faac5942b5ac7b8f4edb57de9b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:03.617930 7035 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b69d6faac5942b5ac7b8f4edb57de9b [term 1 LEADER]: Becoming Leader. State: Replica: 6b69d6faac5942b5ac7b8f4edb57de9b, State: Running, Role: LEADER
I20260924 07:06:03.618237 7035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b69d6faac5942b5ac7b8f4edb57de9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b69d6faac5942b5ac7b8f4edb57de9b" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 39783 } }
I20260924 07:06:03.618934 7030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b69d6faac5942b5ac7b8f4edb57de9b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:06:03.619063 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:03.604429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48320 (local address 127.3.217.62:39783)
0924 07:06:03.604805 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:03.604819 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:03.604852 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:06:03.605513 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:03.605530 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:03.605837 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:03.606098 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:03.606937 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:03.607986 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:03.609072 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:03.609535 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:03.609645 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:03.612782 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:03.612790 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:03.612811 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:03.612855 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:03.615279 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:03.615859 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:03.615867 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:03.615874 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:03.616002 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:03.616707 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:03.616711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:03.616713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:03.617199 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:03.617400 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:03.618049 (+ 649us) server_negotiation.cc:299] Negotiation successful
0924 07:06:03.618308 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":107,"threads_started":1}
I20260924 07:06:03.619402 7035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b69d6faac5942b5ac7b8f4edb57de9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b69d6faac5942b5ac7b8f4edb57de9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b69d6faac5942b5ac7b8f4edb57de9b" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 39783 } } }
I20260924 07:06:03.619540 7035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b69d6faac5942b5ac7b8f4edb57de9b [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:03.619786 7035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b69d6faac5942b5ac7b8f4edb57de9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b69d6faac5942b5ac7b8f4edb57de9b. Latest consensus state: current_term: 1 leader_uuid: "6b69d6faac5942b5ac7b8f4edb57de9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b69d6faac5942b5ac7b8f4edb57de9b" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 39783 } } }
I20260924 07:06:03.619943 7035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b69d6faac5942b5ac7b8f4edb57de9b [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:03.621045 7040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:06:03.623529 7040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:06:03.630208 7040 catalog_manager.cc:1347] Generated new cluster ID: afbf3bfe770a4272858c649ef319db7f
I20260924 07:06:03.630335 7040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:06:03.651873 7040 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:06:03.652794 7040 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:06:03.659715 7040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b69d6faac5942b5ac7b8f4edb57de9b: Generated new TSK 0
I20260924 07:06:03.660305 7040 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
I20260924 07:06:03.691112 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:39783
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:03.799068 7056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:03.799278 7056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:03.799316 7056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:03.803272 7056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:03.803323 7056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:03.803387 7056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:06:03.808465 7056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:39783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:03.809350 7056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:03.810238 7056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:03.818943 7062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:03.818957 7061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:03.818959 7064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:03.819262 7056 server_base.cc:1034] running on GCE node
I20260924 07:06:03.819628 7056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:03.820231 7056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:03.821404 7056 hybrid_clock.cc:648] HybridClock initialized: now 1790233563821390 us; error 24 us; skew 500 ppm
Sep 24 07:06:03 dist-test-slave-2p7j krb5kdc[6949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233563, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:03.824471 7056 init.cc:377] Logged in from keytab as kudu/127.3.217.1@KRBTEST.COM (short username kudu)
I20260924 07:06:03.825310 7056 webserver.cc:460] Webserver started at http://127.3.217.1:35587/ using document root <none> and password file <none>
I20260924 07:06:03.825826 7056 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:03.825901 7056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:03.826084 7056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:03.827986 7056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "a06379f7d72a4f918d72537193bc1894"
format_stamp: "Formatted at 2026-09-24 07:06:03 on dist-test-slave-2p7j"
server_key: "737fec29122668b937b7a33f898ae7e0"
server_key_iv: "123cf3f813c5f117409416e056cb1253"
server_key_version: "encryptionkey@0"
I20260924 07:06:03.828483 7056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "a06379f7d72a4f918d72537193bc1894"
format_stamp: "Formatted at 2026-09-24 07:06:03 on dist-test-slave-2p7j"
server_key: "737fec29122668b937b7a33f898ae7e0"
server_key_iv: "123cf3f813c5f117409416e056cb1253"
server_key_version: "encryptionkey@0"
I20260924 07:06:03.831923 7056 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:06:03.834568 7071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:03.835839 7056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:06:03.835945 7056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "a06379f7d72a4f918d72537193bc1894"
format_stamp: "Formatted at 2026-09-24 07:06:03 on dist-test-slave-2p7j"
server_key: "737fec29122668b937b7a33f898ae7e0"
server_key_iv: "123cf3f813c5f117409416e056cb1253"
server_key_version: "encryptionkey@0"
I20260924 07:06:03.836091 7056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:03.849571 7056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:03.852757 7056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:03.852907 7056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:03.853428 7056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:03.854205 7056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:03.854257 7056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:03.854297 7056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:03.854322 7056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:03.863972 7056 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:42127
I20260924 07:06:03.864013 7184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:42127 every 8 connection(s)
I20260924 07:06:03.864854 7056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:06:03.866571 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 7056
I20260924 07:06:03.866683 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:06:03.866977 3940 external_mini_cluster.cc:1428] Setting key 5955c603380c42931d9d8915a3a0cdca
Sep 24 07:06:03 dist-test-slave-2p7j krb5kdc[6949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233563, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.2@KRBTEST.COM" created.
I20260924 07:06:03.878098 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:03.866523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:40807 (local address 127.3.217.62:39783)
0924 07:06:03.866650 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:03.866655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:03.867556 (+ 901us) server_negotiation.cc:407] Connection header received
0924 07:06:03.869068 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:03.869071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:03.869119 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:03.869186 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:03.870268 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:03.870794 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:03.871935 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:03.872175 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:03.872259 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:03.874611 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:03.874614 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:03.874616 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:03.874638 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:03.876163 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:03.876654 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:03.876657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:03.876659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:03.876730 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:03.877030 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:03.877033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:03.877034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:03.877203 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:03.877288 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:03.877797 (+ 509us) server_negotiation.cc:299] Negotiation successful
0924 07:06:03.877932 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:06:03.878831 7187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:03.866922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:39783 (local address 127.3.217.1:40807)
0924 07:06:03.867410 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:03.867461 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:03.868879 (+ 1418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:03.869339 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:03.869345 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:03.869686 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:03.870134 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:03.870147 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:03.870911 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:03.870914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:03.871815 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:03.871825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:03.872306 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:03.872309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:03.872462 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:03.873078 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:03.873094 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:03.874461 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:03.876281 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:03.876286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:03.876301 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:03.876558 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:03.876811 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:03.876814 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:03.876816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:03.876943 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:03.877311 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:03.877317 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:03.877580 (+ 263us) client_negotiation.cc:769] Sending connection context
0924 07:06:03.877740 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 07:06:03.878063 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":137,"threads_started":1}
I20260924 07:06:03.880070 7185 heartbeater.cc:347] Connected to a master server at 127.3.217.62:39783
I20260924 07:06:03.880470 7185 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:03.881026 7185 heartbeater.cc:511] Master 127.3.217.62:39783 requested a full tablet report, sending...
I20260924 07:06:03.882474 6994 ts_manager.cc:194] Registered new tserver with Master: a06379f7d72a4f918d72537193bc1894 (127.3.217.1:42127)
I20260924 07:06:03.883564 6994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.1@KRBTEST.COM'} at 127.3.217.1:40807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
I20260924 07:06:03.894527 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:39783
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:03.999246 7190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:03.999470 7190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:03.999536 7190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:04.003343 7190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:04.003391 7190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:04.003465 7190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:06:04.008467 7190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:39783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:04.009404 7190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:04.010295 7190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:04.019191 7196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:04.019294 7195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:04.019301 7198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:04.019493 7190 server_base.cc:1034] running on GCE node
I20260924 07:06:04.019941 7190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:04.020529 7190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:04.021714 7190 hybrid_clock.cc:648] HybridClock initialized: now 1790233564021676 us; error 54 us; skew 500 ppm
Sep 24 07:06:04 dist-test-slave-2p7j krb5kdc[6949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233564, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:04.024324 7190 init.cc:377] Logged in from keytab as kudu/127.3.217.2@KRBTEST.COM (short username kudu)
I20260924 07:06:04.025064 7190 webserver.cc:460] Webserver started at http://127.3.217.2:40011/ using document root <none> and password file <none>
I20260924 07:06:04.025537 7190 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:04.025588 7190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:04.025735 7190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:04.027592 7190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "a81e4574f9e5482c9c61e0178d192aa4"
format_stamp: "Formatted at 2026-09-24 07:06:04 on dist-test-slave-2p7j"
server_key: "42baf95acd5a9328dd552ade936fa7cf"
server_key_iv: "bc31e2791182b4fd5b93f9d564b18631"
server_key_version: "encryptionkey@0"
I20260924 07:06:04.028083 7190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "a81e4574f9e5482c9c61e0178d192aa4"
format_stamp: "Formatted at 2026-09-24 07:06:04 on dist-test-slave-2p7j"
server_key: "42baf95acd5a9328dd552ade936fa7cf"
server_key_iv: "bc31e2791182b4fd5b93f9d564b18631"
server_key_version: "encryptionkey@0"
I20260924 07:06:04.031425 7190 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:06:04.033651 7205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:04.035295 7190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:06:04.035398 7190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "a81e4574f9e5482c9c61e0178d192aa4"
format_stamp: "Formatted at 2026-09-24 07:06:04 on dist-test-slave-2p7j"
server_key: "42baf95acd5a9328dd552ade936fa7cf"
server_key_iv: "bc31e2791182b4fd5b93f9d564b18631"
server_key_version: "encryptionkey@0"
I20260924 07:06:04.035494 7190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:04.053009 7190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:04.055660 7190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:04.055817 7190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:04.056326 7190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:04.057050 7190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:04.057153 7190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:04.057257 7190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:04.057293 7190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:04.065979 7190 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:37845
I20260924 07:06:04.066087 7318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:37845 every 8 connection(s)
I20260924 07:06:04.066805 7190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:06:04.070031 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 7190
I20260924 07:06:04.070151 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:06:04.070465 3940 external_mini_cluster.cc:1428] Setting key 6890d370e770b902f77f00f4b9458de5
Sep 24 07:06:04 dist-test-slave-2p7j krb5kdc[6949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233564, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.217.3@KRBTEST.COM; defaulting to no policy
I20260924 07:06:04.082670 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:04.069109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:44233 (local address 127.3.217.62:39783)
0924 07:06:04.069307 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:04.069313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:04.070235 (+ 922us) server_negotiation.cc:407] Connection header received
0924 07:06:04.071793 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:04.071796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:04.071848 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:04.071919 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:04.073355 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:04.074335 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:04.075521 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:04.075767 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:04.075856 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:04.078404 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:04.078408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:04.078410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:04.078437 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:04.079902 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:04.080393 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:04.080396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:04.080397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:04.080468 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:04.081270 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:04.081273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:04.081274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:04.081459 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:04.081543 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:04.082432 (+ 889us) server_negotiation.cc:299] Negotiation successful
0924 07:06:04.082535 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.3@KRBTEST.COM" created.
I20260924 07:06:04.084017 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:04.069444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:39783 (local address 127.3.217.2:44233)
0924 07:06:04.070064 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:04.070119 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:04.071616 (+ 1497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:04.072086 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:04.072094 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:04.072541 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:04.073141 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:04.073158 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:04.074480 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:04.074484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:04.075369 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:04.075378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:04.075900 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:04.075904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:04.076109 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:04.076669 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:04.076684 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:04.078243 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:04.080026 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:04.080032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:04.080044 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:04.080300 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:04.080700 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:04.080704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:04.080706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:04.081123 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:04.081637 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:04.081645 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:04.082052 (+ 407us) client_negotiation.cc:769] Sending connection context
0924 07:06:04.082339 (+ 287us) client_negotiation.cc:240] Negotiation successful
0924 07:06:04.082778 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":169,"threads_started":1}
I20260924 07:06:04.085184 7319 heartbeater.cc:347] Connected to a master server at 127.3.217.62:39783
I20260924 07:06:04.085522 7319 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:04.086269 7319 heartbeater.cc:511] Master 127.3.217.62:39783 requested a full tablet report, sending...
I20260924 07:06:04.087643 6994 ts_manager.cc:194] Registered new tserver with Master: a81e4574f9e5482c9c61e0178d192aa4 (127.3.217.2:37845)
I20260924 07:06:04.088472 6994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.2@KRBTEST.COM'} at 127.3.217.2:44233
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
I20260924 07:06:04.099064 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:39783
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:04.209863 7324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:04.210281 7324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:04.210421 7324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:04.215446 7324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:04.215533 7324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:04.215634 7324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:06:04.220944 7324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:39783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:04.221881 7324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:04.222797 7324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:04.232203 7330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:04.232252 7332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:04.232261 7329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:04.232544 7324 server_base.cc:1034] running on GCE node
I20260924 07:06:04.233012 7324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:04.233563 7324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:04.234858 7324 hybrid_clock.cc:648] HybridClock initialized: now 1790233564234814 us; error 50 us; skew 500 ppm
Sep 24 07:06:04 dist-test-slave-2p7j krb5kdc[6949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233564, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:04.237890 7324 init.cc:377] Logged in from keytab as kudu/127.3.217.3@KRBTEST.COM (short username kudu)
I20260924 07:06:04.238713 7324 webserver.cc:460] Webserver started at http://127.3.217.3:36827/ using document root <none> and password file <none>
I20260924 07:06:04.239274 7324 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:04.239341 7324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:04.239518 7324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:04.241542 7324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "8e608a6f25a54507817799a00e47c828"
format_stamp: "Formatted at 2026-09-24 07:06:04 on dist-test-slave-2p7j"
server_key: "122acc87aecb619a88e36ef7e8d749b7"
server_key_iv: "d3940e820ecad5b7bff22911e1fa1b71"
server_key_version: "encryptionkey@0"
I20260924 07:06:04.242062 7324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "8e608a6f25a54507817799a00e47c828"
format_stamp: "Formatted at 2026-09-24 07:06:04 on dist-test-slave-2p7j"
server_key: "122acc87aecb619a88e36ef7e8d749b7"
server_key_iv: "d3940e820ecad5b7bff22911e1fa1b71"
server_key_version: "encryptionkey@0"
I20260924 07:06:04.245568 7324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 07:06:04.248283 7339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:04.249724 7324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:06:04.249840 7324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "8e608a6f25a54507817799a00e47c828"
format_stamp: "Formatted at 2026-09-24 07:06:04 on dist-test-slave-2p7j"
server_key: "122acc87aecb619a88e36ef7e8d749b7"
server_key_iv: "d3940e820ecad5b7bff22911e1fa1b71"
server_key_version: "encryptionkey@0"
I20260924 07:06:04.249933 7324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:04.266110 7324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:04.268862 7324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:04.269016 7324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:04.269546 7324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:04.270299 7324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:04.270344 7324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:04.270392 7324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:04.270421 7324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:04.279551 7324 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:35481
I20260924 07:06:04.279628 7452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:35481 every 8 connection(s)
I20260924 07:06:04.280400 7324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:06:04.285293 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 7324
I20260924 07:06:04.285408 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:06:04.285651 3940 external_mini_cluster.cc:1428] Setting key 3800e6ad84e14bb0a2c944ddc2fd639d
Sep 24 07:06:04 dist-test-slave-2p7j krb5kdc[6949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233564, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:06:04.294466 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:04.282075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:46061 (local address 127.3.217.62:39783)
0924 07:06:04.282195 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:04.282198 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:04.283198 (+ 1000us) server_negotiation.cc:407] Connection header received
0924 07:06:04.284812 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:04.284815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:04.284860 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:04.284934 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:04.286037 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:04.286634 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:04.287731 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:04.287952 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:04.288030 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:04.290395 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:04.290398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:04.290400 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:04.290421 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:04.292020 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:04.292685 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:04.292688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:04.292690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:04.292773 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:04.293128 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:04.293131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:04.293133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:04.293359 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:04.293452 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:04.294163 (+ 711us) server_negotiation.cc:299] Negotiation successful
0924 07:06:04.294306 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:06:04.295234 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:04.282369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:39783 (local address 127.3.217.3:46061)
0924 07:06:04.282977 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:04.283044 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:04.284577 (+ 1533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:04.285070 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:04.285077 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:04.285406 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:04.285907 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:04.285922 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:04.286746 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:04.286748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:04.287621 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:04.287630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:04.288082 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:04.288086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:04.288232 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:04.288820 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:04.288837 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:04.290242 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:04.292195 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:04.292204 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:04.292225 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:04.292571 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:04.292860 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:04.292863 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:04.292865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:04.293031 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:04.293475 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:04.293483 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:04.293887 (+ 404us) client_negotiation.cc:769] Sending connection context
0924 07:06:04.294077 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 07:06:04.294400 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":111,"threads_started":1}
I20260924 07:06:04.296586 7453 heartbeater.cc:347] Connected to a master server at 127.3.217.62:39783
I20260924 07:06:04.296792 7453 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:04.297245 7453 heartbeater.cc:511] Master 127.3.217.62:39783 requested a full tablet report, sending...
I20260924 07:06:04.298173 6993 ts_manager.cc:194] Registered new tserver with Master: 8e608a6f25a54507817799a00e47c828 (127.3.217.3:35481)
I20260924 07:06:04.298719 6993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.3@KRBTEST.COM'} at 127.3.217.3:46061
I20260924 07:06:04.298781 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 07:06:04 dist-test-slave-2p7j krb5kdc[6949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233564, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 07:06:04 dist-test-slave-2p7j krb5kdc[6949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233564, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:06:04.322707 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:04.314983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48336 (local address 127.3.217.62:39783)
0924 07:06:04.315087 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:04.315091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:04.315237 (+ 146us) server_negotiation.cc:407] Connection header received
0924 07:06:04.315337 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:04.315339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:04.315374 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:04.315444 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:04.316099 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:04.316601 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:04.317445 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:04.317654 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:04.317736 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:04.320004 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:04.320030 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:04.320032 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:04.320054 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:04.321299 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:04.321815 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:04.321818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:04.321819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:04.321899 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:04.322090 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:04.322092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:04.322093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:04.322280 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:04.322380 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:04.322512 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 07:06:04.322597 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 07:06:04.323046 6993 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48336
I20260924 07:06:04.324040 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 7056
I20260924 07:06:04.329958 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 7190
I20260924 07:06:04.334308 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 7324
I20260924 07:06:04.338411 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 6962
2026-09-24T07:06:04Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2053 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:06:04 dist-test-slave-2p7j krb5kdc[7465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:06:04 dist-test-slave-2p7j krb5kdc[7465](info): set up 2 sockets
Sep 24 07:06:04 dist-test-slave-2p7j krb5kdc[7465](info): commencing operation
krb5kdc: starting...
W20260924 07:06:05.300014 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:06:05 dist-test-slave-2p7j krb5kdc[7465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:06:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:06:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:06:05.414860 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:42297
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:42297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:05.524112 7478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:05.524501 7478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:05.524569 7478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:05.530408 7478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:06:05.530501 7478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:05.530527 7478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:05.530550 7478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:06:05.530570 7478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:06:05.536942 7478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:42297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:42297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:05.537845 7478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:05.538739 7478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:05.546890 7484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:05.546929 7486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:05.547044 7478 server_base.cc:1034] running on GCE node
W20260924 07:06:05.546929 7483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:05.547698 7478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:05.548702 7478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:05.549876 7478 hybrid_clock.cc:648] HybridClock initialized: now 1790233565549862 us; error 27 us; skew 500 ppm
Sep 24 07:06:05 dist-test-slave-2p7j krb5kdc[7465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233565, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:05.552440 7478 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:06:05.553201 7478 webserver.cc:460] Webserver started at http://127.3.217.62:41557/ using document root <none> and password file <none>
I20260924 07:06:05.553694 7478 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:05.553757 7478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:05.553947 7478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:05.555842 7478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "8930afd2e2da44edb602c3a499a65f9c"
format_stamp: "Formatted at 2026-09-24 07:06:05 on dist-test-slave-2p7j"
server_key: "ec57ab0c2b75e314b8cbd740bb584d72"
server_key_iv: "0565baf457ff3e0cc22ca774dd792ac1"
server_key_version: "encryptionkey@0"
I20260924 07:06:05.556351 7478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "8930afd2e2da44edb602c3a499a65f9c"
format_stamp: "Formatted at 2026-09-24 07:06:05 on dist-test-slave-2p7j"
server_key: "ec57ab0c2b75e314b8cbd740bb584d72"
server_key_iv: "0565baf457ff3e0cc22ca774dd792ac1"
server_key_version: "encryptionkey@0"
I20260924 07:06:05.559751 7478 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:06:05.562013 7493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:05.563477 7478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:06:05.563624 7478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "8930afd2e2da44edb602c3a499a65f9c"
format_stamp: "Formatted at 2026-09-24 07:06:05 on dist-test-slave-2p7j"
server_key: "ec57ab0c2b75e314b8cbd740bb584d72"
server_key_iv: "0565baf457ff3e0cc22ca774dd792ac1"
server_key_version: "encryptionkey@0"
I20260924 07:06:05.563714 7478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:05.585616 7478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:05.588438 7478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:05.588594 7478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:05.594712 7478 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:42297
I20260924 07:06:05.595149 7545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:42297 every 8 connection(s)
I20260924 07:06:05.595638 7478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:06:05.598145 7546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:05.601116 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 7478
I20260924 07:06:05.601226 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:06:05.601464 3940 external_mini_cluster.cc:1428] Setting key c67d8126015fc93e92e1fd6a91726758
I20260924 07:06:05.603662 7546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8930afd2e2da44edb602c3a499a65f9c: Bootstrap starting.
I20260924 07:06:05.606215 7546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8930afd2e2da44edb602c3a499a65f9c: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:05.607009 7546 log.cc:824] T 00000000000000000000000000000000 P 8930afd2e2da44edb602c3a499a65f9c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:05.608821 7546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8930afd2e2da44edb602c3a499a65f9c: No bootstrap required, opened a new log
Sep 24 07:06:05 dist-test-slave-2p7j krb5kdc[7465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:06:05.612079 7546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8930afd2e2da44edb602c3a499a65f9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8930afd2e2da44edb602c3a499a65f9c" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42297 } }
I20260924 07:06:05.612280 7546 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8930afd2e2da44edb602c3a499a65f9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:05.612344 7546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8930afd2e2da44edb602c3a499a65f9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8930afd2e2da44edb602c3a499a65f9c, State: Initialized, Role: FOLLOWER
I20260924 07:06:05.612736 7546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8930afd2e2da44edb602c3a499a65f9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8930afd2e2da44edb602c3a499a65f9c" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42297 } }
I20260924 07:06:05.612861 7546 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8930afd2e2da44edb602c3a499a65f9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:06:05.612926 7546 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8930afd2e2da44edb602c3a499a65f9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:06:05.613016 7546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8930afd2e2da44edb602c3a499a65f9c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:05.613796 7546 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8930afd2e2da44edb602c3a499a65f9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8930afd2e2da44edb602c3a499a65f9c" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42297 } }
I20260924 07:06:05.614114 7546 leader_election.cc:302] T 00000000000000000000000000000000 P 8930afd2e2da44edb602c3a499a65f9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8930afd2e2da44edb602c3a499a65f9c; no voters:
I20260924 07:06:05.614379 7546 leader_election.cc:288] T 00000000000000000000000000000000 P 8930afd2e2da44edb602c3a499a65f9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:06:05.614583 7551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8930afd2e2da44edb602c3a499a65f9c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:05.614924 7551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8930afd2e2da44edb602c3a499a65f9c [term 1 LEADER]: Becoming Leader. State: Replica: 8930afd2e2da44edb602c3a499a65f9c, State: Running, Role: LEADER
I20260924 07:06:05.615221 7551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8930afd2e2da44edb602c3a499a65f9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8930afd2e2da44edb602c3a499a65f9c" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42297 } }
I20260924 07:06:05.615525 7546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8930afd2e2da44edb602c3a499a65f9c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:06:05.616776 7549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:05.602725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34222 (local address 127.3.217.62:42297)
0924 07:06:05.603108 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:05.603125 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:05.603165 (+ 40us) server_negotiation.cc:407] Connection header received
0924 07:06:05.603968 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:05.603989 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:05.604364 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:05.604700 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:05.605325 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:05.606749 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:05.607653 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:05.608095 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:05.608207 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:05.610802 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:05.610807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:05.610818 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:05.610843 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:05.612947 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:05.613502 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:05.613509 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:05.613513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:05.613645 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:05.613936 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:05.613939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:05.613940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:05.614559 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:05.615546 (+ 987us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:05.615695 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 07:06:05.615989 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":105,"threads_started":1}
I20260924 07:06:05.616757 7553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8930afd2e2da44edb602c3a499a65f9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8930afd2e2da44edb602c3a499a65f9c. Latest consensus state: current_term: 1 leader_uuid: "8930afd2e2da44edb602c3a499a65f9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8930afd2e2da44edb602c3a499a65f9c" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42297 } } }
I20260924 07:06:05.616873 7553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8930afd2e2da44edb602c3a499a65f9c [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:05.617182 7552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8930afd2e2da44edb602c3a499a65f9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8930afd2e2da44edb602c3a499a65f9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8930afd2e2da44edb602c3a499a65f9c" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42297 } } }
I20260924 07:06:05.617268 7552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8930afd2e2da44edb602c3a499a65f9c [sys.catalog]: This master's current role is: LEADER
W20260924 07:06:05.620577 7566 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8930afd2e2da44edb602c3a499a65f9c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:06:05.620653 7566 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:06:05.620738 7562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:06:05.621470 7562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:06:05.626907 7562 catalog_manager.cc:1347] Generated new cluster ID: bb37734cdaf245d8acdd2c80f360dd33
I20260924 07:06:05.626967 7562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:06:05.642607 7562 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:06:05.643487 7562 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:06:05.646968 7562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8930afd2e2da44edb602c3a499a65f9c: Generated new TSK 0
I20260924 07:06:05.647732 7562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
I20260924 07:06:05.675410 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42297
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:05.794940 7572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:05.795190 7572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:05.795245 7572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:05.799185 7572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:05.799244 7572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:05.799321 7572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:06:05.804399 7572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:05.805382 7572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:05.806311 7572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:05.816072 7580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:05.816088 7578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:05.816098 7577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:05.816498 7572 server_base.cc:1034] running on GCE node
I20260924 07:06:05.816895 7572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:05.817708 7572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:05.818984 7572 hybrid_clock.cc:648] HybridClock initialized: now 1790233565818944 us; error 45 us; skew 500 ppm
Sep 24 07:06:05 dist-test-slave-2p7j krb5kdc[7465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233565, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:05.822153 7572 init.cc:377] Logged in from keytab as kudu/127.3.217.1@KRBTEST.COM (short username kudu)
I20260924 07:06:05.822990 7572 webserver.cc:460] Webserver started at http://127.3.217.1:43605/ using document root <none> and password file <none>
I20260924 07:06:05.823565 7572 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:05.823642 7572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:05.823838 7572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:05.825758 7572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "cc9070975aa54ecebcb7baf0634cc5ae"
format_stamp: "Formatted at 2026-09-24 07:06:05 on dist-test-slave-2p7j"
server_key: "d850678b61e536595c90b0c12dead5ce"
server_key_iv: "eb619c6cec5a822cf90b8fbc389fd245"
server_key_version: "encryptionkey@0"
I20260924 07:06:05.826279 7572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "cc9070975aa54ecebcb7baf0634cc5ae"
format_stamp: "Formatted at 2026-09-24 07:06:05 on dist-test-slave-2p7j"
server_key: "d850678b61e536595c90b0c12dead5ce"
server_key_iv: "eb619c6cec5a822cf90b8fbc389fd245"
server_key_version: "encryptionkey@0"
I20260924 07:06:05.830060 7572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 07:06:05.832509 7587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:05.833658 7572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:06:05.833762 7572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "cc9070975aa54ecebcb7baf0634cc5ae"
format_stamp: "Formatted at 2026-09-24 07:06:05 on dist-test-slave-2p7j"
server_key: "d850678b61e536595c90b0c12dead5ce"
server_key_iv: "eb619c6cec5a822cf90b8fbc389fd245"
server_key_version: "encryptionkey@0"
I20260924 07:06:05.833853 7572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:05.871410 7572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:05.874377 7572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:05.874549 7572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:05.875115 7572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:05.875955 7572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:05.876003 7572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:05.876053 7572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:05.876082 7572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:05.884346 7572 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:37391
I20260924 07:06:05.884395 7700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:37391 every 8 connection(s)
I20260924 07:06:05.885174 7572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:06:05.892488 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 7572
I20260924 07:06:05.892707 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:06:05.893146 3940 external_mini_cluster.cc:1428] Setting key f27a4da14bcf1c7376ba9aeb07c0ffe4
Sep 24 07:06:05 dist-test-slave-2p7j krb5kdc[7465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233565, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:06:05.902077 7549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:05.887113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:33361 (local address 127.3.217.62:42297)
0924 07:06:05.887300 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:05.887305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:05.888350 (+ 1045us) server_negotiation.cc:407] Connection header received
0924 07:06:05.890046 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:05.890049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:05.890095 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:05.890180 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:05.891614 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:05.892156 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:05.894152 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:05.894456 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:05.894567 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:05.898006 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:05.898011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:05.898014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:05.898055 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:05.899820 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:05.900409 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:05.900413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:05.900415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:05.900505 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:05.900877 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:05.900881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:05.900883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:05.901110 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:05.901209 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:05.901713 (+ 504us) server_negotiation.cc:299] Negotiation successful
0924 07:06:05.901838 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 07:06:05.902982 7703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:05.887434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42297 (local address 127.3.217.1:33361)
0924 07:06:05.888128 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:05.888198 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:05.889832 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:05.890463 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:05.890473 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:05.890953 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:05.891422 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:05.891439 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:05.892321 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:05.892326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:05.893945 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:05.893961 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:05.894643 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:05.894650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:05.894909 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:05.895580 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:05.895602 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:05.897728 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:05.899965 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:05.899970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:05.899981 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:05.900289 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:05.900613 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:05.900616 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:05.900617 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:05.900755 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:05.901222 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:05.901227 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:05.901475 (+ 248us) client_negotiation.cc:769] Sending connection context
0924 07:06:05.901625 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 07:06:05.901895 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.3.217.2@KRBTEST.COM; defaulting to no policy
I20260924 07:06:05.904212 7701 heartbeater.cc:347] Connected to a master server at 127.3.217.62:42297
I20260924 07:06:05.904462 7701 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:05.904898 7701 heartbeater.cc:511] Master 127.3.217.62:42297 requested a full tablet report, sending...
I20260924 07:06:05.906280 7510 ts_manager.cc:194] Registered new tserver with Master: cc9070975aa54ecebcb7baf0634cc5ae (127.3.217.1:37391)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.2@KRBTEST.COM" created.
I20260924 07:06:05.907426 7510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.1@KRBTEST.COM'} at 127.3.217.1:33361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
I20260924 07:06:05.922035 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42297
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:06.026853 7706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:06.027082 7706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:06.027122 7706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:06.032239 7706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:06.032302 7706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:06.032358 7706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:06:06.037632 7706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:06.038561 7706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:06.039479 7706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:06.048137 7712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:06.048144 7711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:06.048138 7714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:06.048525 7706 server_base.cc:1034] running on GCE node
I20260924 07:06:06.048847 7706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:06.049377 7706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:06.050552 7706 hybrid_clock.cc:648] HybridClock initialized: now 1790233566050538 us; error 30 us; skew 500 ppm
Sep 24 07:06:06 dist-test-slave-2p7j krb5kdc[7465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:06.053009 7706 init.cc:377] Logged in from keytab as kudu/127.3.217.2@KRBTEST.COM (short username kudu)
I20260924 07:06:06.053759 7706 webserver.cc:460] Webserver started at http://127.3.217.2:43143/ using document root <none> and password file <none>
I20260924 07:06:06.054253 7706 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:06.054311 7706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:06.054476 7706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:06.056368 7706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "c6c9a6bb6f8a4a288c5ffe72512d69c3"
format_stamp: "Formatted at 2026-09-24 07:06:06 on dist-test-slave-2p7j"
server_key: "ac3623409041f8c3451608f406f97553"
server_key_iv: "b3b865f7ba6fe662351dc90d9cf7108d"
server_key_version: "encryptionkey@0"
I20260924 07:06:06.056855 7706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "c6c9a6bb6f8a4a288c5ffe72512d69c3"
format_stamp: "Formatted at 2026-09-24 07:06:06 on dist-test-slave-2p7j"
server_key: "ac3623409041f8c3451608f406f97553"
server_key_iv: "b3b865f7ba6fe662351dc90d9cf7108d"
server_key_version: "encryptionkey@0"
I20260924 07:06:06.060286 7706 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:06:06.062583 7721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:06.063915 7706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:06:06.064015 7706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "c6c9a6bb6f8a4a288c5ffe72512d69c3"
format_stamp: "Formatted at 2026-09-24 07:06:06 on dist-test-slave-2p7j"
server_key: "ac3623409041f8c3451608f406f97553"
server_key_iv: "b3b865f7ba6fe662351dc90d9cf7108d"
server_key_version: "encryptionkey@0"
I20260924 07:06:06.064113 7706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:06.086074 7706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:06.089018 7706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:06.089222 7706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:06.089818 7706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:06.090689 7706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:06.090771 7706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:06.090827 7706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:06.090859 7706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:06.100091 7706 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:42465
I20260924 07:06:06.100268 7834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:42465 every 8 connection(s)
I20260924 07:06:06.100960 7706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:06:06.107749 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 7706
I20260924 07:06:06.107863 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:06:06.108134 3940 external_mini_cluster.cc:1428] Setting key 861c096aba6bd2e96f3c22de2cd35f79
Sep 24 07:06:06 dist-test-slave-2p7j krb5kdc[7465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:06:06.115252 7549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:06.102683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:45161 (local address 127.3.217.62:42297)
0924 07:06:06.102848 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:06.102853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:06.103992 (+ 1139us) server_negotiation.cc:407] Connection header received
0924 07:06:06.105370 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:06.105373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:06.105416 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:06.105489 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:06.106547 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:06.107078 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:06.108304 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:06.108522 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:06.108611 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:06.111114 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:06.111117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:06.111119 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:06.111139 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:06.112784 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:06.113633 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:06.113637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:06.113639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:06.113724 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:06.114178 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:06.114181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:06.114183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:06.114401 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:06.114501 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:06.114973 (+ 472us) server_negotiation.cc:299] Negotiation successful
0924 07:06:06.115090 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 07:06:06.116127 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:06.103254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42297 (local address 127.3.217.2:45161)
0924 07:06:06.103860 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:06.103900 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:06.105159 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:06.105628 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:06.105634 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:06.105952 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:06.106405 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:06.106419 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:06.107200 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:06.107203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:06.108196 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:06.108207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:06.108798 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:06.108801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:06.108932 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:06.109474 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:06.109490 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:06.110954 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:06.113013 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:06.113022 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:06.113040 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:06.113500 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:06.113903 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:06.113907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:06.113909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:06.114075 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:06.114514 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:06.114520 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:06.114744 (+ 224us) client_negotiation.cc:769] Sending connection context
0924 07:06:06.114907 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 07:06:06.115209 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":136,"threads_started":1}
WARNING: no policy specified for kudu/127.3.217.3@KRBTEST.COM; defaulting to no policy
I20260924 07:06:06.117352 7835 heartbeater.cc:347] Connected to a master server at 127.3.217.62:42297
I20260924 07:06:06.117746 7835 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:06.118297 7835 heartbeater.cc:511] Master 127.3.217.62:42297 requested a full tablet report, sending...
I20260924 07:06:06.119447 7510 ts_manager.cc:194] Registered new tserver with Master: c6c9a6bb6f8a4a288c5ffe72512d69c3 (127.3.217.2:42465)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.3@KRBTEST.COM" created.
I20260924 07:06:06.120460 7510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.2@KRBTEST.COM'} at 127.3.217.2:45161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
I20260924 07:06:06.135246 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42297
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:06.249743 7840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:06.250016 7840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:06.250061 7840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:06.254282 7840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:06.254346 7840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:06.254411 7840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:06:06.259956 7840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:06.260986 7840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:06.261969 7840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:06.271328 7848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:06.271335 7845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:06.271351 7846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:06.271534 7840 server_base.cc:1034] running on GCE node
I20260924 07:06:06.271943 7840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:06.272548 7840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:06.273804 7840 hybrid_clock.cc:648] HybridClock initialized: now 1790233566273768 us; error 29 us; skew 500 ppm
Sep 24 07:06:06 dist-test-slave-2p7j krb5kdc[7465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:06.277400 7840 init.cc:377] Logged in from keytab as kudu/127.3.217.3@KRBTEST.COM (short username kudu)
I20260924 07:06:06.278229 7840 webserver.cc:460] Webserver started at http://127.3.217.3:34275/ using document root <none> and password file <none>
I20260924 07:06:06.278740 7840 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:06.278802 7840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:06.278963 7840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:06.280964 7840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "f84963ca4a1a4e9885e2bd97bca147ad"
format_stamp: "Formatted at 2026-09-24 07:06:06 on dist-test-slave-2p7j"
server_key: "3912952eeb60b8ce7080717957c0e39f"
server_key_iv: "170c486956ad1bb959f0fc0608a29104"
server_key_version: "encryptionkey@0"
I20260924 07:06:06.281476 7840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "f84963ca4a1a4e9885e2bd97bca147ad"
format_stamp: "Formatted at 2026-09-24 07:06:06 on dist-test-slave-2p7j"
server_key: "3912952eeb60b8ce7080717957c0e39f"
server_key_iv: "170c486956ad1bb959f0fc0608a29104"
server_key_version: "encryptionkey@0"
I20260924 07:06:06.285164 7840 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:06:06.287721 7855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:06.289429 7840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:06:06.289630 7840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "f84963ca4a1a4e9885e2bd97bca147ad"
format_stamp: "Formatted at 2026-09-24 07:06:06 on dist-test-slave-2p7j"
server_key: "3912952eeb60b8ce7080717957c0e39f"
server_key_iv: "170c486956ad1bb959f0fc0608a29104"
server_key_version: "encryptionkey@0"
I20260924 07:06:06.289772 7840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:06.305498 7840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:06.308571 7840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:06.308765 7840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:06.309325 7840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:06.310127 7840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:06.310178 7840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:06.310235 7840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:06.310268 7840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:06.321000 7840 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:43967
I20260924 07:06:06.321012 7968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:43967 every 8 connection(s)
I20260924 07:06:06.322492 7840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:06:06.331784 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 7840
I20260924 07:06:06.331972 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:06:06.332355 3940 external_mini_cluster.cc:1428] Setting key 1338bf04c14a92e45aaa5b537deac9b5
Sep 24 07:06:06 dist-test-slave-2p7j krb5kdc[7465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:06:06.337291 7549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:06.324275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:46559 (local address 127.3.217.62:42297)
0924 07:06:06.324431 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:06.324436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:06.325247 (+ 811us) server_negotiation.cc:407] Connection header received
0924 07:06:06.326544 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:06.326548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:06.326595 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:06.326802 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:06.327970 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:06.328503 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:06.329727 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:06.330001 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:06.330133 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:06.333237 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:06.333242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:06.333245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:06.333279 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:06.335040 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:06.335617 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:06.335620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:06.335622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:06.335720 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:06.336279 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:06.336281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:06.336282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:06.336491 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:06.336571 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:06.337057 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 07:06:06.337172 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 07:06:06.338150 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:06.324635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42297 (local address 127.3.217.3:46559)
0924 07:06:06.325100 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:06.325140 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:06.326343 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:06.326924 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:06.326931 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:06.327262 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:06.327808 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:06.327821 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:06.328613 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:06.328616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:06.329556 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:06.329565 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:06.330247 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:06.330251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:06.330458 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:06.331192 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:06.331211 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:06.333048 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:06.335184 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:06.335190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:06.335200 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:06.335484 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:06.335866 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:06.335870 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:06.335872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:06.336140 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:06.336616 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:06.336621 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:06.336856 (+ 235us) client_negotiation.cc:769] Sending connection context
0924 07:06:06.337007 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 07:06:06.337245 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":105,"threads_started":1}
I20260924 07:06:06.339201 7969 heartbeater.cc:347] Connected to a master server at 127.3.217.62:42297
I20260924 07:06:06.339418 7969 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:06.339900 7969 heartbeater.cc:511] Master 127.3.217.62:42297 requested a full tablet report, sending...
I20260924 07:06:06.340847 7510 ts_manager.cc:194] Registered new tserver with Master: f84963ca4a1a4e9885e2bd97bca147ad (127.3.217.3:43967)
I20260924 07:06:06.341421 7510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.3@KRBTEST.COM'} at 127.3.217.3:46559
I20260924 07:06:06.346688 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 07:06:06 dist-test-slave-2p7j krb5kdc[7465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 07:06:06 dist-test-slave-2p7j krb5kdc[7465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.1@KRBTEST.COM
I20260924 07:06:06.371832 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:06.362119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48110 (local address 127.3.217.1:37391)
0924 07:06:06.362423 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:06.362438 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:06.362460 (+ 22us) server_negotiation.cc:407] Connection header received
0924 07:06:06.362507 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:06.362512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:06.362727 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:06.362847 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:06.363607 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:06.364388 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:06.365198 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:06.365567 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:06.365658 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:06.368002 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:06.368011 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:06.368022 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:06.368046 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:06.369563 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:06.370242 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:06.370247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:06.370248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:06.370360 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:06.370725 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:06.370728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:06.370730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:06.370980 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:06.371165 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:06.371459 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 07:06:06.371674 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":106,"threads_started":1}
Sep 24 07:06:06 dist-test-slave-2p7j krb5kdc[7465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:06:06.383366 7549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:06.374922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35852 (local address 127.3.217.62:42297)
0924 07:06:06.375051 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:06.375054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:06.375177 (+ 123us) server_negotiation.cc:407] Connection header received
0924 07:06:06.375248 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:06.375250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:06.375316 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:06.375408 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:06.375985 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:06.376519 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:06.377286 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:06.377507 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:06.377586 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:06.379789 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:06.379793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:06.379796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:06.379821 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:06.381318 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:06.381992 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:06.381995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:06.381996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:06.382068 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:06.382512 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:06.382522 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:06.382523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:06.382731 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:06.382832 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:06.383105 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 07:06:06.383198 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 07:06:06.383788 7510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35852
I20260924 07:06:06.384582 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 7572
I20260924 07:06:06.390058 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 7706
I20260924 07:06:06.395673 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 7840
I20260924 07:06:06.401012 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 7478
2026-09-24T07:06:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2063 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:06:06 dist-test-slave-2p7j krb5kdc[7981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:06:06 dist-test-slave-2p7j krb5kdc[7981](info): set up 2 sockets
Sep 24 07:06:06 dist-test-slave-2p7j krb5kdc[7981](info): commencing operation
krb5kdc: starting...
W20260924 07:06:07.384285 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:06:07 dist-test-slave-2p7j krb5kdc[7981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:06:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:06:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:06:07.510911 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:45843
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:39153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:45843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:07.623992 7994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:07.624244 7994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:07.624302 7994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:07.628700 7994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:06:07.628759 7994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:07.628798 7994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:07.628821 7994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:06:07.628855 7994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:06:07.633976 7994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:39153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:45843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:45843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:07.634907 7994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:07.635844 7994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:07.644127 7999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:07.644119 8000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:07.644114 8002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:07.644374 7994 server_base.cc:1034] running on GCE node
I20260924 07:06:07.644922 7994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:07.645921 7994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:07.647118 7994 hybrid_clock.cc:648] HybridClock initialized: now 1790233567647097 us; error 58 us; skew 500 ppm
Sep 24 07:06:07 dist-test-slave-2p7j krb5kdc[7981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233567, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:07.656190 7994 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:06:07.656926 7994 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:07.657001 7994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:07.657234 7994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:07.659241 7994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "c50e7f15247a4df69f4f4843329c32fb"
format_stamp: "Formatted at 2026-09-24 07:06:07 on dist-test-slave-2p7j"
server_key: "42d994aae59ed9ea35c52041ccac386f"
server_key_iv: "6b76269d5bc9e82ce1d28e9fb8370a0f"
server_key_version: "encryptionkey@0"
I20260924 07:06:07.659811 7994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "c50e7f15247a4df69f4f4843329c32fb"
format_stamp: "Formatted at 2026-09-24 07:06:07 on dist-test-slave-2p7j"
server_key: "42d994aae59ed9ea35c52041ccac386f"
server_key_iv: "6b76269d5bc9e82ce1d28e9fb8370a0f"
server_key_version: "encryptionkey@0"
I20260924 07:06:07.664445 7994 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 07:06:07.666980 8008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:07.668488 7994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:06:07.668605 7994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "c50e7f15247a4df69f4f4843329c32fb"
format_stamp: "Formatted at 2026-09-24 07:06:07 on dist-test-slave-2p7j"
server_key: "42d994aae59ed9ea35c52041ccac386f"
server_key_iv: "6b76269d5bc9e82ce1d28e9fb8370a0f"
server_key_version: "encryptionkey@0"
I20260924 07:06:07.668702 7994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:07.689265 7994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:07.692135 7994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:07.692312 7994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:07.699203 7994 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:45843
I20260924 07:06:07.699461 8060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:45843 every 8 connection(s)
I20260924 07:06:07.699818 7994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:06:07.702418 8061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:07.707726 8061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c50e7f15247a4df69f4f4843329c32fb: Bootstrap starting.
I20260924 07:06:07.708061 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 7994
I20260924 07:06:07.708168 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:06:07.708885 3940 external_mini_cluster.cc:1428] Setting key 68f3be80cfb4f3c01fef0a6be6861245
I20260924 07:06:07.710624 8061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c50e7f15247a4df69f4f4843329c32fb: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:07.711341 8061 log.cc:824] T 00000000000000000000000000000000 P c50e7f15247a4df69f4f4843329c32fb: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:07.713372 8061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c50e7f15247a4df69f4f4843329c32fb: No bootstrap required, opened a new log
I20260924 07:06:07.717780 8061 raft_consensus.cc:348] T 00000000000000000000000000000000 P c50e7f15247a4df69f4f4843329c32fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c50e7f15247a4df69f4f4843329c32fb" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45843 } }
Sep 24 07:06:07 dist-test-slave-2p7j krb5kdc[7981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:06:07.718065 8061 raft_consensus.cc:374] T 00000000000000000000000000000000 P c50e7f15247a4df69f4f4843329c32fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:07.718125 8061 raft_consensus.cc:729] T 00000000000000000000000000000000 P c50e7f15247a4df69f4f4843329c32fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c50e7f15247a4df69f4f4843329c32fb, State: Initialized, Role: FOLLOWER
I20260924 07:06:07.718598 8061 consensus_queue.cc:260] T 00000000000000000000000000000000 P c50e7f15247a4df69f4f4843329c32fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c50e7f15247a4df69f4f4843329c32fb" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45843 } }
I20260924 07:06:07.718778 8061 raft_consensus.cc:388] T 00000000000000000000000000000000 P c50e7f15247a4df69f4f4843329c32fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:06:07.718855 8061 raft_consensus.cc:482] T 00000000000000000000000000000000 P c50e7f15247a4df69f4f4843329c32fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:06:07.718966 8061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c50e7f15247a4df69f4f4843329c32fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:07.719915 8061 raft_consensus.cc:504] T 00000000000000000000000000000000 P c50e7f15247a4df69f4f4843329c32fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c50e7f15247a4df69f4f4843329c32fb" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45843 } }
I20260924 07:06:07.720230 8061 leader_election.cc:302] T 00000000000000000000000000000000 P c50e7f15247a4df69f4f4843329c32fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c50e7f15247a4df69f4f4843329c32fb; no voters:
I20260924 07:06:07.720537 8061 leader_election.cc:288] T 00000000000000000000000000000000 P c50e7f15247a4df69f4f4843329c32fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:06:07.720628 8066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c50e7f15247a4df69f4f4843329c32fb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:07.720803 8066 raft_consensus.cc:686] T 00000000000000000000000000000000 P c50e7f15247a4df69f4f4843329c32fb [term 1 LEADER]: Becoming Leader. State: Replica: c50e7f15247a4df69f4f4843329c32fb, State: Running, Role: LEADER
I20260924 07:06:07.721139 8066 consensus_queue.cc:237] T 00000000000000000000000000000000 P c50e7f15247a4df69f4f4843329c32fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c50e7f15247a4df69f4f4843329c32fb" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45843 } }
I20260924 07:06:07.721818 8061 sys_catalog.cc:564] T 00000000000000000000000000000000 P c50e7f15247a4df69f4f4843329c32fb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:06:07.723138 8068 sys_catalog.cc:455] T 00000000000000000000000000000000 P c50e7f15247a4df69f4f4843329c32fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c50e7f15247a4df69f4f4843329c32fb. Latest consensus state: current_term: 1 leader_uuid: "c50e7f15247a4df69f4f4843329c32fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c50e7f15247a4df69f4f4843329c32fb" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45843 } } }
I20260924 07:06:07.723158 8067 sys_catalog.cc:455] T 00000000000000000000000000000000 P c50e7f15247a4df69f4f4843329c32fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c50e7f15247a4df69f4f4843329c32fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c50e7f15247a4df69f4f4843329c32fb" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45843 } } }
I20260924 07:06:07.723284 8067 sys_catalog.cc:458] T 00000000000000000000000000000000 P c50e7f15247a4df69f4f4843329c32fb [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:07.723284 8068 sys_catalog.cc:458] T 00000000000000000000000000000000 P c50e7f15247a4df69f4f4843329c32fb [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:07.723466 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:07.710111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46644 (local address 127.3.217.62:45843)
0924 07:06:07.710515 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:07.710530 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:07.710566 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:06:07.711238 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:07.711255 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:07.711626 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:07.711899 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:07.712679 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:07.714054 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:07.715021 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:07.715766 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:07.715932 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:07.718596 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:07.718603 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:07.718619 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:07.718652 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:07.720287 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:07.720683 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:07.720689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:07.720695 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:07.720853 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:07.721147 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:07.721150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:07.721152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:07.721558 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:07.721729 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:07.721998 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 07:06:07.722278 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":113,"threads_started":1}
I20260924 07:06:07.724185 8075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:06:07.727619 8075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:06:07.732594 8075 catalog_manager.cc:1347] Generated new cluster ID: 76ce5d559d2045eba0b5c53e45c189b6
I20260924 07:06:07.732658 8075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:06:07.759101 8075 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:06:07.760136 8075 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:06:07.770080 8075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c50e7f15247a4df69f4f4843329c32fb: Generated new TSK 0
I20260924 07:06:07.771129 8075 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
I20260924 07:06:07.811580 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:45843
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:39153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:07.923516 8087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:07.923745 8087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:07.923787 8087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:07.928368 8087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:07.928440 8087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:07.928507 8087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:06:07.934556 8087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:39153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:45843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:07.935472 8087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:07.936540 8087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:06:07.946573 8087 server_base.cc:1034] running on GCE node
W20260924 07:06:07.946521 8095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:07.946542 8093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:07.946527 8092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:07.947052 8087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:07.947674 8087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:07.948871 8087 hybrid_clock.cc:648] HybridClock initialized: now 1790233567948859 us; error 43 us; skew 500 ppm
Sep 24 07:06:07 dist-test-slave-2p7j krb5kdc[7981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233567, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:07.951437 8087 init.cc:377] Logged in from keytab as kudu/127.3.217.1@KRBTEST.COM (short username kudu)
I20260924 07:06:07.952111 8087 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:07.952175 8087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:07.952327 8087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:07.954161 8087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "75bba093363b471e887699860d37b1a4"
format_stamp: "Formatted at 2026-09-24 07:06:07 on dist-test-slave-2p7j"
server_key: "0ad5608ce6e5b5a423aa1b3743795453"
server_key_iv: "e2c064b442c11bc411076d2ec3210d83"
server_key_version: "encryptionkey@0"
I20260924 07:06:07.954643 8087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "75bba093363b471e887699860d37b1a4"
format_stamp: "Formatted at 2026-09-24 07:06:07 on dist-test-slave-2p7j"
server_key: "0ad5608ce6e5b5a423aa1b3743795453"
server_key_iv: "e2c064b442c11bc411076d2ec3210d83"
server_key_version: "encryptionkey@0"
I20260924 07:06:07.958427 8087 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 07:06:07.961229 8101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:07.962579 8087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:06:07.962683 8087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "75bba093363b471e887699860d37b1a4"
format_stamp: "Formatted at 2026-09-24 07:06:07 on dist-test-slave-2p7j"
server_key: "0ad5608ce6e5b5a423aa1b3743795453"
server_key_iv: "e2c064b442c11bc411076d2ec3210d83"
server_key_version: "encryptionkey@0"
I20260924 07:06:07.962811 8087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:07.982484 8087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:07.985611 8087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:07.985791 8087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:07.986359 8087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:07.987150 8087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:07.987196 8087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:07.987253 8087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:07.987278 8087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:07.995525 8087 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:43601
I20260924 07:06:07.995589 8214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:43601 every 8 connection(s)
I20260924 07:06:07.996069 8087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:06:07.998129 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 8087
I20260924 07:06:07.998253 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:06:07.998754 3940 external_mini_cluster.cc:1428] Setting key 20ff4aa6cccf9f8e0980311d69537e79
Sep 24 07:06:08 dist-test-slave-2p7j krb5kdc[7981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233567, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.217.2@KRBTEST.COM; defaulting to no policy
I20260924 07:06:08.011898 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:07.997809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:59179 (local address 127.3.217.62:45843)
0924 07:06:07.997967 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:07.997971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:07.998801 (+ 830us) server_negotiation.cc:407] Connection header received
0924 07:06:08.000319 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:08.000324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:08.000376 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:08.000447 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:08.001683 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.002624 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.004288 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.004611 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.004716 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:08.007603 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:08.007607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:08.007610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:08.007637 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:08.009585 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:08.010089 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:08.010092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:08.010093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:08.010172 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:08.010750 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:08.010753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:08.010754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:08.010953 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:08.011031 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:08.011620 (+ 589us) server_negotiation.cc:299] Negotiation successful
0924 07:06:08.011770 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.2@KRBTEST.COM" created.
I20260924 07:06:08.013008 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:07.998166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:45843 (local address 127.3.217.1:59179)
0924 07:06:07.998661 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:07.998702 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:08.000059 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:08.000638 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:08.000645 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:08.000991 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:08.001459 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:08.001473 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.002783 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.002787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:08.004148 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:08.004159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.004794 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.004798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:08.004985 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:08.005742 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:08.005762 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:08.007398 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:08.009728 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:08.009733 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:08.009752 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:08.009996 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:08.010396 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:08.010402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:08.010405 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:08.010643 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:08.011066 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:08.011072 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:08.011350 (+ 278us) client_negotiation.cc:769] Sending connection context
0924 07:06:08.011535 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 07:06:08.011835 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":134,"threads_started":1}
I20260924 07:06:08.014350 8215 heartbeater.cc:347] Connected to a master server at 127.3.217.62:45843
I20260924 07:06:08.014611 8215 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:08.015261 8215 heartbeater.cc:511] Master 127.3.217.62:45843 requested a full tablet report, sending...
I20260924 07:06:08.016904 8025 ts_manager.cc:194] Registered new tserver with Master: 75bba093363b471e887699860d37b1a4 (127.3.217.1:43601)
I20260924 07:06:08.018009 8025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.1@KRBTEST.COM'} at 127.3.217.1:59179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
I20260924 07:06:08.031924 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:45843
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:39153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:08.139531 8220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:08.139797 8220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:08.139858 8220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:08.144160 8220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:08.144224 8220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:08.144304 8220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:06:08.149511 8220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:39153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:45843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:08.150566 8220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:08.151578 8220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:08.160748 8225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:08.160751 8226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:08.160737 8228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:08.161062 8220 server_base.cc:1034] running on GCE node
I20260924 07:06:08.161406 8220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:08.162112 8220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:08.163322 8220 hybrid_clock.cc:648] HybridClock initialized: now 1790233568163281 us; error 74 us; skew 500 ppm
Sep 24 07:06:08 dist-test-slave-2p7j krb5kdc[7981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233568, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:08.166000 8220 init.cc:377] Logged in from keytab as kudu/127.3.217.2@KRBTEST.COM (short username kudu)
I20260924 07:06:08.166751 8220 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:08.166832 8220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:08.166985 8220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:08.168880 8220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "9d2896986e954635aa3bdd6f1f7212ef"
format_stamp: "Formatted at 2026-09-24 07:06:08 on dist-test-slave-2p7j"
server_key: "8ce8e6fb5f88e9cdadbe7c1fc4a7b0d6"
server_key_iv: "5c6cdf443a72757e5d27696744fc8664"
server_key_version: "encryptionkey@0"
I20260924 07:06:08.169359 8220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "9d2896986e954635aa3bdd6f1f7212ef"
format_stamp: "Formatted at 2026-09-24 07:06:08 on dist-test-slave-2p7j"
server_key: "8ce8e6fb5f88e9cdadbe7c1fc4a7b0d6"
server_key_iv: "5c6cdf443a72757e5d27696744fc8664"
server_key_version: "encryptionkey@0"
I20260924 07:06:08.173447 8220 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:06:08.176844 8234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:08.178123 8220 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:06:08.178222 8220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "9d2896986e954635aa3bdd6f1f7212ef"
format_stamp: "Formatted at 2026-09-24 07:06:08 on dist-test-slave-2p7j"
server_key: "8ce8e6fb5f88e9cdadbe7c1fc4a7b0d6"
server_key_iv: "5c6cdf443a72757e5d27696744fc8664"
server_key_version: "encryptionkey@0"
I20260924 07:06:08.178313 8220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:08.194733 8220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:08.198078 8220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:08.198246 8220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:08.198843 8220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:08.199721 8220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:08.199780 8220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:08.199824 8220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:08.199842 8220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:08.210073 8220 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:34051
I20260924 07:06:08.210095 8347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:34051 every 8 connection(s)
I20260924 07:06:08.210763 8220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:06:08.218786 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 8220
I20260924 07:06:08.218922 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:06:08.219244 3940 external_mini_cluster.cc:1428] Setting key a6c2ccd175a2c3e787945635ee8d9afc
Sep 24 07:06:08 dist-test-slave-2p7j krb5kdc[7981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233568, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:06:08.225526 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:08.212409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:40861 (local address 127.3.217.62:45843)
0924 07:06:08.212544 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:08.212548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:08.213475 (+ 927us) server_negotiation.cc:407] Connection header received
0924 07:06:08.214622 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:08.214625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:08.214671 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:08.214743 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:08.215895 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.216438 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.217530 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.217754 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.217836 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:08.220453 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:08.220457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:08.220460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:08.220485 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:08.222083 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:08.223416 (+ 1333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:08.223420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:08.223422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:08.223551 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:08.224073 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:08.224080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:08.224083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:08.224495 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:08.224642 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:08.225293 (+ 651us) server_negotiation.cc:299] Negotiation successful
0924 07:06:08.225407 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:06:08.226517 8350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:08.212782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:45843 (local address 127.3.217.2:40861)
0924 07:06:08.213236 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:08.213276 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:08.214436 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:08.214878 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:08.214885 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:08.215220 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:08.215748 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:08.215762 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.216550 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.216553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:08.217407 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:08.217416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.217885 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.217889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:08.218026 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:08.218543 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:08.218563 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:08.220278 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:08.223052 (+ 2774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:08.223059 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:08.223071 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:08.223324 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:08.223653 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:08.223656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:08.223658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:08.223825 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:08.224662 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:08.224669 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:08.224985 (+ 316us) client_negotiation.cc:769] Sending connection context
0924 07:06:08.225200 (+ 215us) client_negotiation.cc:240] Negotiation successful
0924 07:06:08.225519 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":114,"threads_started":1}
I20260924 07:06:08.227952 8348 heartbeater.cc:347] Connected to a master server at 127.3.217.62:45843
I20260924 07:06:08.228196 8348 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:08.228686 8348 heartbeater.cc:511] Master 127.3.217.62:45843 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.217.3@KRBTEST.COM; defaulting to no policy
I20260924 07:06:08.229729 8025 ts_manager.cc:194] Registered new tserver with Master: 9d2896986e954635aa3bdd6f1f7212ef (127.3.217.2:34051)
I20260924 07:06:08.230324 8025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.2@KRBTEST.COM'} at 127.3.217.2:40861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
I20260924 07:06:08.247797 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:45843
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:39153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:08.360998 8353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:08.361230 8353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:08.361274 8353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:08.365322 8353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:08.365375 8353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:08.365439 8353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:06:08.370671 8353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:39153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:45843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:08.371644 8353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:08.372591 8353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:08.381412 8361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:08.381501 8353 server_base.cc:1034] running on GCE node
W20260924 07:06:08.381564 8358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:08.381417 8359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:08.382082 8353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:08.382718 8353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:08.383889 8353 hybrid_clock.cc:648] HybridClock initialized: now 1790233568383865 us; error 37 us; skew 500 ppm
Sep 24 07:06:08 dist-test-slave-2p7j krb5kdc[7981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233568, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:08.386899 8353 init.cc:377] Logged in from keytab as kudu/127.3.217.3@KRBTEST.COM (short username kudu)
I20260924 07:06:08.387676 8353 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:08.387745 8353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:08.387939 8353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:08.389848 8353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "e83478fdab6345b18594989373109c6a"
format_stamp: "Formatted at 2026-09-24 07:06:08 on dist-test-slave-2p7j"
server_key: "df8564e9a98bbf0b6eee298460c457bd"
server_key_iv: "4279997e3ea9f4436447edfe5c970ee2"
server_key_version: "encryptionkey@0"
I20260924 07:06:08.390347 8353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "e83478fdab6345b18594989373109c6a"
format_stamp: "Formatted at 2026-09-24 07:06:08 on dist-test-slave-2p7j"
server_key: "df8564e9a98bbf0b6eee298460c457bd"
server_key_iv: "4279997e3ea9f4436447edfe5c970ee2"
server_key_version: "encryptionkey@0"
I20260924 07:06:08.393750 8353 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 07:06:08.396096 8367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:08.397281 8353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:06:08.397382 8353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "e83478fdab6345b18594989373109c6a"
format_stamp: "Formatted at 2026-09-24 07:06:08 on dist-test-slave-2p7j"
server_key: "df8564e9a98bbf0b6eee298460c457bd"
server_key_iv: "4279997e3ea9f4436447edfe5c970ee2"
server_key_version: "encryptionkey@0"
I20260924 07:06:08.397467 8353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:08.418308 8353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:08.423135 8353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:08.423365 8353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:08.424270 8353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:08.425443 8353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:08.425561 8353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:08.425637 8353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:08.425668 8353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:08.438015 8353 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:42069
I20260924 07:06:08.438087 8480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:42069 every 8 connection(s)
I20260924 07:06:08.438666 8353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:06:08.445001 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 8353
I20260924 07:06:08.445122 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:06:08.445439 3940 external_mini_cluster.cc:1428] Setting key f5af4ec383a1952144c403ae4aee7d97
Sep 24 07:06:08 dist-test-slave-2p7j krb5kdc[7981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233568, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:06:08.454703 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:08.440342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:46277 (local address 127.3.217.62:45843)
0924 07:06:08.440468 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:08.440471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:08.441248 (+ 777us) server_negotiation.cc:407] Connection header received
0924 07:06:08.442551 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:08.442555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:08.442607 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:08.442682 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:08.444351 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.444974 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.446445 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.446729 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.447030 (+ 301us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:08.450468 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:08.450471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:08.450473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:08.450501 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:08.452183 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:08.452765 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:08.452768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:08.452769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:08.453018 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:08.453569 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:08.453572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:08.453573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:08.453787 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:08.453897 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:08.454455 (+ 558us) server_negotiation.cc:299] Negotiation successful
0924 07:06:08.454578 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:06:08.455420 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:08.440631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:45843 (local address 127.3.217.3:46277)
0924 07:06:08.441102 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:08.441141 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:08.442365 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:08.442841 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:08.442847 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:08.443402 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:08.444151 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:08.444165 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.445148 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.445151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:08.446330 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:08.446341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.447088 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.447094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:08.447386 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:08.448149 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:08.448169 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:08.450268 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:08.452321 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:08.452326 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:08.452338 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:08.452626 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:08.453193 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:08.453196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:08.453198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:08.453421 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:08.453953 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:08.453958 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:08.454197 (+ 239us) client_negotiation.cc:769] Sending connection context
0924 07:06:08.454366 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 07:06:08.454629 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":113,"threads_started":1}
I20260924 07:06:08.456555 8481 heartbeater.cc:347] Connected to a master server at 127.3.217.62:45843
I20260924 07:06:08.456790 8481 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:08.457204 8481 heartbeater.cc:511] Master 127.3.217.62:45843 requested a full tablet report, sending...
I20260924 07:06:08.458419 8025 ts_manager.cc:194] Registered new tserver with Master: e83478fdab6345b18594989373109c6a (127.3.217.3:42069)
I20260924 07:06:08.459131 8025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.3@KRBTEST.COM'} at 127.3.217.3:46277
I20260924 07:06:08.460052 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:06:08.469955 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:08.462105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46648 (local address 127.3.217.62:45843)
0924 07:06:08.462299 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:08.462303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:08.462317 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:06:08.462400 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:08.462402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:08.462437 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:08.462500 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:08.463221 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.463813 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.464748 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.464988 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.465073 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:08.466357 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:08.466360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:08.466362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:08.466382 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:08.467722 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:08.468535 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:08.468596 (+ 61us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:08.468599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:08.468685 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:08.469002 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:08.469004 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:08.469005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:08.469216 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:08.469332 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:08.469732 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 07:06:08.469842 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260924 07:06:08.474123 8025 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:06:08.478022 8025 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:06:08.493294 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:08.487757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:42069 (local address 127.0.0.1:58332)
0924 07:06:08.488017 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:08.488030 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:08.488116 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:08.488831 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:08.488834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:08.488849 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:08.488952 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:08.488958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.490239 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.490243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:08.491796 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:08.491806 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.492737 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.492741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:08.492829 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:08.492863 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 07:06:08.492988 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 07:06:08.493122 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":258,"thread_start_us":128,"threads_started":1}
I20260924 07:06:08.493330 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:08.487950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58332 (local address 127.3.217.3:42069)
0924 07:06:08.488319 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:08.488329 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:08.488347 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:06:08.488416 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:08.488419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:08.488572 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:08.488699 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:08.489066 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.490045 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.491923 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.492558 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.492665 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:08.492799 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:08.493052 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 07:06:08.493152 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":153,"threads_started":1}
I20260924 07:06:08.493746 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:08.486551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:34051 (local address 127.0.0.1:44178)
0924 07:06:08.487660 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:08.487672 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:08.487756 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:08.488255 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:08.488261 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:08.488281 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:08.488422 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:08.488430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.489927 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.489932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:08.492152 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:08.492173 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.493340 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.493343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:08.493448 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:08.493471 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:06:08.493537 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 07:06:08.493613 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"mutex_wait_us":254,"thread_start_us":118,"threads_started":1}
I20260924 07:06:08.494099 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:08.486489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:43601 (local address 127.0.0.1:35152)
0924 07:06:08.486787 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:08.486830 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:08.486946 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:08.488891 (+ 1945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:08.488894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:08.488921 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:08.489015 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:08.489021 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.490261 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.490264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:08.492169 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:08.492175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.493744 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.493747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:08.493833 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:08.493856 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:06:08.493905 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 07:06:08.493981 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":331,"thread_start_us":64,"threads_started":1}
I20260924 07:06:08.494113 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:08.487108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44178 (local address 127.3.217.2:34051)
0924 07:06:08.487396 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:08.487405 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:08.487848 (+ 443us) server_negotiation.cc:407] Connection header received
0924 07:06:08.487929 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:08.487935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:08.488116 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:08.488266 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:08.488544 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.489707 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.492332 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.493070 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.493169 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:08.493282 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:08.493891 (+ 609us) server_negotiation.cc:299] Negotiation successful
0924 07:06:08.493981 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":95,"threads_started":1}
I20260924 07:06:08.496049 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:08.486643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35152 (local address 127.3.217.1:43601)
0924 07:06:08.487771 (+ 1128us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:08.487777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:08.487800 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:06:08.488559 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:08.488564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:08.488731 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:08.489142 (+ 411us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:08.489275 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.490139 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.492871 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.493622 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.495650 (+ 2028us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:08.495774 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:08.495821 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 07:06:08.495903 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1036,"thread_start_us":98,"threads_started":1}
I20260924 07:06:08.496613 8282 tablet_service.cc:1467] Processing CreateTablet for tablet 33032b002d6d4c2e861bfe5f8fdaaec5 (DEFAULT_TABLE table=test-table [id=66fb156b22dd4404b3e5b30a4ee819a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:08.496616 8415 tablet_service.cc:1467] Processing CreateTablet for tablet 33032b002d6d4c2e861bfe5f8fdaaec5 (DEFAULT_TABLE table=test-table [id=66fb156b22dd4404b3e5b30a4ee819a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:08.497489 8282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33032b002d6d4c2e861bfe5f8fdaaec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:08.497502 8415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33032b002d6d4c2e861bfe5f8fdaaec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:08.504632 8498 tablet_bootstrap.cc:492] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a: Bootstrap starting.
I20260924 07:06:08.504688 8149 tablet_service.cc:1467] Processing CreateTablet for tablet 33032b002d6d4c2e861bfe5f8fdaaec5 (DEFAULT_TABLE table=test-table [id=66fb156b22dd4404b3e5b30a4ee819a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:08.505290 8499 tablet_bootstrap.cc:492] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 9d2896986e954635aa3bdd6f1f7212ef: Bootstrap starting.
I20260924 07:06:08.505817 8149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33032b002d6d4c2e861bfe5f8fdaaec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:08.507170 8498 tablet_bootstrap.cc:654] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:08.507539 8499 tablet_bootstrap.cc:654] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 9d2896986e954635aa3bdd6f1f7212ef: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:08.508638 8498 log.cc:824] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:08.508658 8499 log.cc:824] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 9d2896986e954635aa3bdd6f1f7212ef: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:08.510700 8499 tablet_bootstrap.cc:492] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 9d2896986e954635aa3bdd6f1f7212ef: No bootstrap required, opened a new log
I20260924 07:06:08.510839 8498 tablet_bootstrap.cc:492] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a: No bootstrap required, opened a new log
I20260924 07:06:08.510908 8499 ts_tablet_manager.cc:1397] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 9d2896986e954635aa3bdd6f1f7212ef: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:06:08.510994 8498 ts_tablet_manager.cc:1397] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260924 07:06:08.514072 8502 tablet_bootstrap.cc:492] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 75bba093363b471e887699860d37b1a4: Bootstrap starting.
I20260924 07:06:08.514942 8498 raft_consensus.cc:348] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bba093363b471e887699860d37b1a4" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 43601 } } peers { permanent_uuid: "9d2896986e954635aa3bdd6f1f7212ef" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 34051 } } peers { permanent_uuid: "e83478fdab6345b18594989373109c6a" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 42069 } }
I20260924 07:06:08.514942 8499 raft_consensus.cc:348] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 9d2896986e954635aa3bdd6f1f7212ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bba093363b471e887699860d37b1a4" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 43601 } } peers { permanent_uuid: "9d2896986e954635aa3bdd6f1f7212ef" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 34051 } } peers { permanent_uuid: "e83478fdab6345b18594989373109c6a" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 42069 } }
I20260924 07:06:08.515236 8499 raft_consensus.cc:374] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 9d2896986e954635aa3bdd6f1f7212ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:08.515236 8498 raft_consensus.cc:374] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:08.515298 8498 raft_consensus.cc:729] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e83478fdab6345b18594989373109c6a, State: Initialized, Role: FOLLOWER
I20260924 07:06:08.515298 8499 raft_consensus.cc:729] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 9d2896986e954635aa3bdd6f1f7212ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2896986e954635aa3bdd6f1f7212ef, State: Initialized, Role: FOLLOWER
I20260924 07:06:08.515780 8498 consensus_queue.cc:260] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bba093363b471e887699860d37b1a4" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 43601 } } peers { permanent_uuid: "9d2896986e954635aa3bdd6f1f7212ef" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 34051 } } peers { permanent_uuid: "e83478fdab6345b18594989373109c6a" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 42069 } }
I20260924 07:06:08.515780 8499 consensus_queue.cc:260] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 9d2896986e954635aa3bdd6f1f7212ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bba093363b471e887699860d37b1a4" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 43601 } } peers { permanent_uuid: "9d2896986e954635aa3bdd6f1f7212ef" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 34051 } } peers { permanent_uuid: "e83478fdab6345b18594989373109c6a" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 42069 } }
I20260924 07:06:08.516096 8502 tablet_bootstrap.cc:654] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 75bba093363b471e887699860d37b1a4: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:08.516786 8498 ts_tablet_manager.cc:1428] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 07:06:08.516786 8499 ts_tablet_manager.cc:1428] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 9d2896986e954635aa3bdd6f1f7212ef: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260924 07:06:08.517390 8348 heartbeater.cc:503] Master 127.3.217.62:45843 was elected leader, sending a full tablet report...
I20260924 07:06:08.517704 8502 log.cc:824] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 75bba093363b471e887699860d37b1a4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:08.518224 8481 heartbeater.cc:503] Master 127.3.217.62:45843 was elected leader, sending a full tablet report...
I20260924 07:06:08.519801 8502 tablet_bootstrap.cc:492] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 75bba093363b471e887699860d37b1a4: No bootstrap required, opened a new log
I20260924 07:06:08.520015 8502 ts_tablet_manager.cc:1397] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 75bba093363b471e887699860d37b1a4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:06:08.523382 8502 raft_consensus.cc:348] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 75bba093363b471e887699860d37b1a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bba093363b471e887699860d37b1a4" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 43601 } } peers { permanent_uuid: "9d2896986e954635aa3bdd6f1f7212ef" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 34051 } } peers { permanent_uuid: "e83478fdab6345b18594989373109c6a" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 42069 } }
I20260924 07:06:08.523636 8502 raft_consensus.cc:374] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 75bba093363b471e887699860d37b1a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:08.523676 8502 raft_consensus.cc:729] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 75bba093363b471e887699860d37b1a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75bba093363b471e887699860d37b1a4, State: Initialized, Role: FOLLOWER
I20260924 07:06:08.524142 8502 consensus_queue.cc:260] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 75bba093363b471e887699860d37b1a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bba093363b471e887699860d37b1a4" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 43601 } } peers { permanent_uuid: "9d2896986e954635aa3bdd6f1f7212ef" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 34051 } } peers { permanent_uuid: "e83478fdab6345b18594989373109c6a" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 42069 } }
I20260924 07:06:08.524763 8215 heartbeater.cc:503] Master 127.3.217.62:45843 was elected leader, sending a full tablet report...
I20260924 07:06:08.525077 8502 ts_tablet_manager.cc:1428] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 75bba093363b471e887699860d37b1a4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 07:06:08.590052 8504 raft_consensus.cc:482] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:06:08.590246 8504 raft_consensus.cc:504] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bba093363b471e887699860d37b1a4" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 43601 } } peers { permanent_uuid: "9d2896986e954635aa3bdd6f1f7212ef" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 34051 } } peers { permanent_uuid: "e83478fdab6345b18594989373109c6a" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 42069 } }
I20260924 07:06:08.591349 8504 leader_election.cc:288] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75bba093363b471e887699860d37b1a4 (127.3.217.1:43601), 9d2896986e954635aa3bdd6f1f7212ef (127.3.217.2:34051)
I20260924 07:06:08.595081 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:08.591971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:53873 (local address 127.3.217.1:43601)
0924 07:06:08.592088 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:08.592092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:08.592231 (+ 139us) server_negotiation.cc:407] Connection header received
0924 07:06:08.592341 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:08.592343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:08.592391 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:08.592450 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:08.592674 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.593174 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.594195 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.594587 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.594673 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:08.594732 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:08.594902 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 07:06:08.594965 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 07:06:08.595096 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:08.591767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:43601 (local address 127.3.217.3:53873)
0924 07:06:08.592156 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:08.592165 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:08.592243 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:08.592472 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:08.592474 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:08.592488 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:08.592588 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:08.592592 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.593271 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.593274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:08.594078 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:08.594086 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.594693 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.594696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:08.594788 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:08.594816 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 07:06:08.594878 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 07:06:08.594949 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":18,"thread_start_us":119,"threads_started":1}
I20260924 07:06:08.595329 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:08.591671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:34051 (local address 127.3.217.3:41889)
0924 07:06:08.591793 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:08.591839 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:08.591927 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:08.592161 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:08.592164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:08.592181 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:08.592286 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:08.592296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.592972 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.592975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:08.593761 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:08.593769 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.594769 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.594774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:08.594919 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:08.594940 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:06:08.595017 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 07:06:08.595212 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":2}
I20260924 07:06:08.595691 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:08.591802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:41889 (local address 127.3.217.2:34051)
0924 07:06:08.591921 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:08.591925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:08.591952 (+ 27us) server_negotiation.cc:407] Connection header received
0924 07:06:08.592011 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:08.592014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:08.592062 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:08.592133 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:08.592394 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.592870 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.593880 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.594472 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.594710 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:08.594790 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:08.595132 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 07:06:08.595583 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:06:08.595933 8169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33032b002d6d4c2e861bfe5f8fdaaec5" candidate_uuid: "e83478fdab6345b18594989373109c6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75bba093363b471e887699860d37b1a4" is_pre_election: true
I20260924 07:06:08.596195 8169 raft_consensus.cc:2454] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 75bba093363b471e887699860d37b1a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e83478fdab6345b18594989373109c6a in term 0.
I20260924 07:06:08.596172 8302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33032b002d6d4c2e861bfe5f8fdaaec5" candidate_uuid: "e83478fdab6345b18594989373109c6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d2896986e954635aa3bdd6f1f7212ef" is_pre_election: true
I20260924 07:06:08.596405 8302 raft_consensus.cc:2454] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 9d2896986e954635aa3bdd6f1f7212ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e83478fdab6345b18594989373109c6a in term 0.
I20260924 07:06:08.596645 8371 leader_election.cc:302] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75bba093363b471e887699860d37b1a4, e83478fdab6345b18594989373109c6a; no voters:
I20260924 07:06:08.596881 8504 raft_consensus.cc:2781] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:06:08.596946 8504 raft_consensus.cc:482] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:06:08.596972 8504 raft_consensus.cc:3037] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:08.597919 8504 raft_consensus.cc:504] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bba093363b471e887699860d37b1a4" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 43601 } } peers { permanent_uuid: "9d2896986e954635aa3bdd6f1f7212ef" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 34051 } } peers { permanent_uuid: "e83478fdab6345b18594989373109c6a" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 42069 } }
I20260924 07:06:08.598232 8504 leader_election.cc:288] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a [CANDIDATE]: Term 1 election: Requested vote from peers 75bba093363b471e887699860d37b1a4 (127.3.217.1:43601), 9d2896986e954635aa3bdd6f1f7212ef (127.3.217.2:34051)
I20260924 07:06:08.598696 8302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33032b002d6d4c2e861bfe5f8fdaaec5" candidate_uuid: "e83478fdab6345b18594989373109c6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d2896986e954635aa3bdd6f1f7212ef"
I20260924 07:06:08.598796 8302 raft_consensus.cc:3037] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 9d2896986e954635aa3bdd6f1f7212ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:08.598924 8169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33032b002d6d4c2e861bfe5f8fdaaec5" candidate_uuid: "e83478fdab6345b18594989373109c6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75bba093363b471e887699860d37b1a4"
I20260924 07:06:08.599115 8169 raft_consensus.cc:3037] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 75bba093363b471e887699860d37b1a4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:08.599857 8302 raft_consensus.cc:2454] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 9d2896986e954635aa3bdd6f1f7212ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e83478fdab6345b18594989373109c6a in term 1.
I20260924 07:06:08.600124 8368 leader_election.cc:302] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d2896986e954635aa3bdd6f1f7212ef, e83478fdab6345b18594989373109c6a; no voters:
I20260924 07:06:08.600286 8504 raft_consensus.cc:2781] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:08.600481 8504 raft_consensus.cc:686] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a [term 1 LEADER]: Becoming Leader. State: Replica: e83478fdab6345b18594989373109c6a, State: Running, Role: LEADER
I20260924 07:06:08.600593 8169 raft_consensus.cc:2454] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 75bba093363b471e887699860d37b1a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e83478fdab6345b18594989373109c6a in term 1.
I20260924 07:06:08.600793 8504 consensus_queue.cc:237] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bba093363b471e887699860d37b1a4" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 43601 } } peers { permanent_uuid: "9d2896986e954635aa3bdd6f1f7212ef" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 34051 } } peers { permanent_uuid: "e83478fdab6345b18594989373109c6a" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 42069 } }
I20260924 07:06:08.603883 8025 catalog_manager.cc:5495] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a reported cstate change: term changed from 0 to 1, leader changed from <none> to e83478fdab6345b18594989373109c6a (127.3.217.3). New cstate: current_term: 1 leader_uuid: "e83478fdab6345b18594989373109c6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bba093363b471e887699860d37b1a4" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 43601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d2896986e954635aa3bdd6f1f7212ef" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 34051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e83478fdab6345b18594989373109c6a" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 42069 } health_report { overall_health: HEALTHY } } }
I20260924 07:06:08.616174 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:08.612735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58340 (local address 127.3.217.3:42069)
0924 07:06:08.612869 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:08.612873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:08.612886 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:06:08.612959 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:08.612962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:08.613009 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:08.613093 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:06:08.613317 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.613790 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.614575 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:08.614773 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:08.614852 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:08.615105 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:06:08.615479 (+ 374us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:06:08.615605 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:08.615809 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 07:06:08.615921 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 07:06:08.621645 8302 raft_consensus.cc:1261] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 9d2896986e954635aa3bdd6f1f7212ef [term 1 FOLLOWER]: Refusing update from remote peer e83478fdab6345b18594989373109c6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:06:08.621896 8169 raft_consensus.cc:1261] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 75bba093363b471e887699860d37b1a4 [term 1 FOLLOWER]: Refusing update from remote peer e83478fdab6345b18594989373109c6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:06:08.622327 8508 consensus_queue.cc:1035] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d2896986e954635aa3bdd6f1f7212ef" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 34051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:06:08.622705 8504 consensus_queue.cc:1035] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bba093363b471e887699860d37b1a4" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 43601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:06:08.629244 8511 mvcc.cc:205] Tried to move back new op lower bound from 7332796697067274240 to 7332796696991535104. Current Snapshot: MvccSnapshot[applied={T|T < 7332796697067274240}]
I20260924 07:06:08.630107 8512 mvcc.cc:205] Tried to move back new op lower bound from 7332796697067274240 to 7332796696991535104. Current Snapshot: MvccSnapshot[applied={T|T < 7332796697067274240}]
I20260924 07:06:08.631940 8513 mvcc.cc:205] Tried to move back new op lower bound from 7332796697067274240 to 7332796696991535104. Current Snapshot: MvccSnapshot[applied={T|T < 7332796697067274240}]
I20260924 07:06:08.640458 8025 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:06:08.640611 8025 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:06:08.643237 8025 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c50e7f15247a4df69f4f4843329c32fb: Sending DeleteTablet for 3 replicas of tablet 33032b002d6d4c2e861bfe5f8fdaaec5
I20260924 07:06:08.644068 8149 tablet_service.cc:1514] Processing DeleteTablet for tablet 33032b002d6d4c2e861bfe5f8fdaaec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:08 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:35152
I20260924 07:06:08.644078 8282 tablet_service.cc:1514] Processing DeleteTablet for tablet 33032b002d6d4c2e861bfe5f8fdaaec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:08 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:44178
I20260924 07:06:08.644537 8520 tablet_replica.cc:330] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 9d2896986e954635aa3bdd6f1f7212ef: stopping tablet replica
I20260924 07:06:08.644660 8415 tablet_service.cc:1514] Processing DeleteTablet for tablet 33032b002d6d4c2e861bfe5f8fdaaec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:08 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:58332
I20260924 07:06:08.644732 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 8087
I20260924 07:06:08.644922 8520 raft_consensus.cc:2229] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 9d2896986e954635aa3bdd6f1f7212ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:06:08.645236 8520 raft_consensus.cc:2258] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 9d2896986e954635aa3bdd6f1f7212ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:08.645356 8522 tablet_replica.cc:330] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a: stopping tablet replica
I20260924 07:06:08.645598 8522 raft_consensus.cc:2229] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:06:08.645942 8522 raft_consensus.cc:2258] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:08.646960 8520 ts_tablet_manager.cc:1905] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 9d2896986e954635aa3bdd6f1f7212ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:06:08.647063 8522 ts_tablet_manager.cc:1905] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:06:08.649624 8522 ts_tablet_manager.cc:1918] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:06:08.649624 8520 ts_tablet_manager.cc:1918] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 9d2896986e954635aa3bdd6f1f7212ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:06:08.649736 8522 log.cc:1196] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/33032b002d6d4c2e861bfe5f8fdaaec5
I20260924 07:06:08.649850 8520 log.cc:1196] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 9d2896986e954635aa3bdd6f1f7212ef: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/33032b002d6d4c2e861bfe5f8fdaaec5
I20260924 07:06:08.650077 8522 ts_tablet_manager.cc:1939] T 33032b002d6d4c2e861bfe5f8fdaaec5 P e83478fdab6345b18594989373109c6a: Deleting consensus metadata
I20260924 07:06:08.650287 8520 ts_tablet_manager.cc:1939] T 33032b002d6d4c2e861bfe5f8fdaaec5 P 9d2896986e954635aa3bdd6f1f7212ef: Deleting consensus metadata
I20260924 07:06:08.651262 8009 catalog_manager.cc:4841] TS e83478fdab6345b18594989373109c6a (127.3.217.3:42069): tablet 33032b002d6d4c2e861bfe5f8fdaaec5 (table test-table [id=66fb156b22dd4404b3e5b30a4ee819a4]) successfully deleted
I20260924 07:06:08.651472 8009 catalog_manager.cc:4841] TS 9d2896986e954635aa3bdd6f1f7212ef (127.3.217.2:34051): tablet 33032b002d6d4c2e861bfe5f8fdaaec5 (table test-table [id=66fb156b22dd4404b3e5b30a4ee819a4]) successfully deleted
W20260924 07:06:08.653666 8012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.217.1:43601 (error 108)
I20260924 07:06:08.653901 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 8220
I20260924 07:06:08.656018 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:08.655441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:43601 (local address 127.0.0.1:35154)
0924 07:06:08.655714 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:08.655877 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.1:43601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 07:06:08.656345 8012 catalog_manager.cc:4584] TS 75bba093363b471e887699860d37b1a4 (127.3.217.1:43601): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 33032b002d6d4c2e861bfe5f8fdaaec5: Network error: Client connection negotiation failed: client connection to 127.3.217.1:43601: connect: Connection refused (error 111)
I20260924 07:06:08.659113 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 8353
I20260924 07:06:08.663882 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 7994
2026-09-24T07:06:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2262 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T07:06:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:06:08Z Disabled control of system clock
I20260924 07:06:08.691651 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:35141
--webserver_interface=127.3.217.62
--webserver_port=0
--builtin_ntp_servers=127.3.217.20:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:35141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 07:06:08.799431 8527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:08.799813 8527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:08.799872 8527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:08.804266 8527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:06:08.804337 8527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:08.804352 8527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:08.804368 8527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:06:08.804382 8527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:06:08.809203 8527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:35141
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:35141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:08.810242 8527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:08.811578 8527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:08.819550 8535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:08.819555 8533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:08.819550 8532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:08.819617 8527 server_base.cc:1034] running on GCE node
I20260924 07:06:08.820343 8527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:08.821368 8527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:08.824653 8527 hybrid_clock.cc:648] HybridClock initialized: now 1790233568824620 us; error 40 us; skew 500 ppm
I20260924 07:06:08.826545 8527 webserver.cc:460] Webserver started at http://127.3.217.62:44259/ using document root <none> and password file <none>
I20260924 07:06:08.827147 8527 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:08.827217 8527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:08.827414 8527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:08.829342 8527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "5268e16f2d624cf09304a39881db75a4"
format_stamp: "Formatted at 2026-09-24 07:06:08 on dist-test-slave-2p7j"
server_key: "8339d35660f6d6be97027ec935afa825"
server_key_iv: "8f6a490cd1de94d75f2f8b99a9dac19d"
server_key_version: "encryptionkey@0"
I20260924 07:06:08.829843 8527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "5268e16f2d624cf09304a39881db75a4"
format_stamp: "Formatted at 2026-09-24 07:06:08 on dist-test-slave-2p7j"
server_key: "8339d35660f6d6be97027ec935afa825"
server_key_iv: "8f6a490cd1de94d75f2f8b99a9dac19d"
server_key_version: "encryptionkey@0"
I20260924 07:06:08.833503 8527 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:06:08.835872 8541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:08.837749 8527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:06:08.837914 8527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "5268e16f2d624cf09304a39881db75a4"
format_stamp: "Formatted at 2026-09-24 07:06:08 on dist-test-slave-2p7j"
server_key: "8339d35660f6d6be97027ec935afa825"
server_key_iv: "8f6a490cd1de94d75f2f8b99a9dac19d"
server_key_version: "encryptionkey@0"
I20260924 07:06:08.838006 8527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:08.844347 8527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:08.844988 8527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:08.845147 8527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:08.851495 8527 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:35141
I20260924 07:06:08.851541 8593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:35141 every 8 connection(s)
I20260924 07:06:08.852432 8527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:06:08.855453 8594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:08.856976 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 8527
I20260924 07:06:08.857106 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:06:08.857438 3940 external_mini_cluster.cc:1428] Setting key a913f97c4adcfc94bd2854e31f85820f
I20260924 07:06:08.862339 8594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5268e16f2d624cf09304a39881db75a4: Bootstrap starting.
I20260924 07:06:08.863889 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:08.859108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36744 (local address 127.3.217.62:35141)
0924 07:06:08.859822 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:08.859838 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:08.859874 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:06:08.860606 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:08.860622 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:08.860959 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:08.861228 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:08.862062 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:08.862067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:08.862077 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:06:08.862087 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:08.862143 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:06:08.862158 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:08.862267 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:08.862358 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 07:06:08.862582 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":130,"threads_started":1}
I20260924 07:06:08.864555 8594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5268e16f2d624cf09304a39881db75a4: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:08.865218 8594 log.cc:824] T 00000000000000000000000000000000 P 5268e16f2d624cf09304a39881db75a4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:08.866914 8594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5268e16f2d624cf09304a39881db75a4: No bootstrap required, opened a new log
I20260924 07:06:08.870265 8594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5268e16f2d624cf09304a39881db75a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5268e16f2d624cf09304a39881db75a4" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 35141 } }
I20260924 07:06:08.870468 8594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5268e16f2d624cf09304a39881db75a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:08.870530 8594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5268e16f2d624cf09304a39881db75a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5268e16f2d624cf09304a39881db75a4, State: Initialized, Role: FOLLOWER
I20260924 07:06:08.870880 8594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5268e16f2d624cf09304a39881db75a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5268e16f2d624cf09304a39881db75a4" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 35141 } }
I20260924 07:06:08.870999 8594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5268e16f2d624cf09304a39881db75a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:06:08.871057 8594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5268e16f2d624cf09304a39881db75a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:06:08.871148 8594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5268e16f2d624cf09304a39881db75a4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:08.871984 8594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5268e16f2d624cf09304a39881db75a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5268e16f2d624cf09304a39881db75a4" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 35141 } }
I20260924 07:06:08.872259 8594 leader_election.cc:302] T 00000000000000000000000000000000 P 5268e16f2d624cf09304a39881db75a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5268e16f2d624cf09304a39881db75a4; no voters:
I20260924 07:06:08.872562 8594 leader_election.cc:288] T 00000000000000000000000000000000 P 5268e16f2d624cf09304a39881db75a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:06:08.872697 8599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5268e16f2d624cf09304a39881db75a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:08.872898 8599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5268e16f2d624cf09304a39881db75a4 [term 1 LEADER]: Becoming Leader. State: Replica: 5268e16f2d624cf09304a39881db75a4, State: Running, Role: LEADER
I20260924 07:06:08.873175 8599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5268e16f2d624cf09304a39881db75a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5268e16f2d624cf09304a39881db75a4" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 35141 } }
I20260924 07:06:08.873716 8594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5268e16f2d624cf09304a39881db75a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:06:08.874306 8601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5268e16f2d624cf09304a39881db75a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5268e16f2d624cf09304a39881db75a4. Latest consensus state: current_term: 1 leader_uuid: "5268e16f2d624cf09304a39881db75a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5268e16f2d624cf09304a39881db75a4" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 35141 } } }
I20260924 07:06:08.874432 8601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5268e16f2d624cf09304a39881db75a4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:08.874825 8600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5268e16f2d624cf09304a39881db75a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5268e16f2d624cf09304a39881db75a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5268e16f2d624cf09304a39881db75a4" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 35141 } } }
I20260924 07:06:08.874923 8600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5268e16f2d624cf09304a39881db75a4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:08.875833 8611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:06:08.878355 8611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:06:08.883306 8611 catalog_manager.cc:1347] Generated new cluster ID: 37c3a4e6037743bf8174fd7b0ce8ff45
I20260924 07:06:08.883380 8611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:06:08.916402 8611 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:06:08.916929 8611 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:06:08.928939 8611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5268e16f2d624cf09304a39881db75a4: Generated new TSK 0
I20260924 07:06:08.929616 8611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:06:08.935017 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:35141
--builtin_ntp_servers=127.3.217.20:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 07:06:09.063022 8618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:09.063280 8618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:09.063320 8618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:09.068221 8618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:09.068288 8618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:09.068346 8618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:06:09.074014 8618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:35141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:09.075760 8618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:09.077632 8618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:09.087039 8626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:09.087093 8618 server_base.cc:1034] running on GCE node
W20260924 07:06:09.087157 8623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:09.087186 8624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:09.087929 8618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:09.089078 8618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:09.090292 8618 hybrid_clock.cc:648] HybridClock initialized: now 1790233569090282 us; error 29 us; skew 500 ppm
I20260924 07:06:09.092757 8618 webserver.cc:460] Webserver started at http://127.3.217.1:38165/ using document root <none> and password file <none>
I20260924 07:06:09.093848 8618 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:09.093942 8618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:09.094203 8618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:09.096544 8618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "cae33adbb3344ff99b0ef44e943eaaa0"
format_stamp: "Formatted at 2026-09-24 07:06:09 on dist-test-slave-2p7j"
server_key: "583faf9e81efe2172840aa56f2be7d1c"
server_key_iv: "8b5bfaf294e2b8d1ca1305f14b1b6aa7"
server_key_version: "encryptionkey@0"
I20260924 07:06:09.097074 8618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "cae33adbb3344ff99b0ef44e943eaaa0"
format_stamp: "Formatted at 2026-09-24 07:06:09 on dist-test-slave-2p7j"
server_key: "583faf9e81efe2172840aa56f2be7d1c"
server_key_iv: "8b5bfaf294e2b8d1ca1305f14b1b6aa7"
server_key_version: "encryptionkey@0"
I20260924 07:06:09.100706 8618 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:06:09.103716 8632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:09.105274 8618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 07:06:09.105522 8618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "cae33adbb3344ff99b0ef44e943eaaa0"
format_stamp: "Formatted at 2026-09-24 07:06:09 on dist-test-slave-2p7j"
server_key: "583faf9e81efe2172840aa56f2be7d1c"
server_key_iv: "8b5bfaf294e2b8d1ca1305f14b1b6aa7"
server_key_version: "encryptionkey@0"
I20260924 07:06:09.105707 8618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:09.111413 8618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:09.112046 8618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:09.112210 8618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:09.112735 8618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:09.113601 8618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:09.113647 8618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:09.113695 8618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:09.113724 8618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:09.124264 8618 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:39821
I20260924 07:06:09.124300 8745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:39821 every 8 connection(s)
I20260924 07:06:09.125224 8618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:06:09.131815 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:09.127660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:60791 (local address 127.3.217.62:35141)
0924 07:06:09.127799 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:09.127802 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:09.128720 (+ 918us) server_negotiation.cc:407] Connection header received
0924 07:06:09.130146 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:09.130149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:09.130196 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:09.130269 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:09.131351 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:09.131355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:09.131357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:06:09.131370 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:09.131389 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:06:09.131398 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:09.131477 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:09.131636 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 07:06:09.131678 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:06:09.132658 8748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:09.127895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:35141 (local address 127.3.217.1:60791)
0924 07:06:09.128558 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:09.128598 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:09.129921 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:09.130455 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:09.130462 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:09.130550 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:09.131101 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:06:09.131103 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:09.131141 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:06:09.131208 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:09.131468 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:09.131476 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:09.131481 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:06:09.131672 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 07:06:09.131908 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":111,"threads_started":1}
I20260924 07:06:09.133950 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 8618
I20260924 07:06:09.134073 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:06:09.134131 8746 heartbeater.cc:347] Connected to a master server at 127.3.217.62:35141
I20260924 07:06:09.134334 8746 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:09.134341 3940 external_mini_cluster.cc:1428] Setting key 721585b4abc5c83d026a807cd8945736
I20260924 07:06:09.134694 8746 heartbeater.cc:511] Master 127.3.217.62:35141 requested a full tablet report, sending...
I20260924 07:06:09.136314 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:35141
--builtin_ntp_servers=127.3.217.20:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 07:06:09.136677 8558 ts_manager.cc:194] Registered new tserver with Master: cae33adbb3344ff99b0ef44e943eaaa0 (127.3.217.1:39821)
W20260924 07:06:09.247488 8749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:09.247767 8749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:09.247821 8749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:09.252306 8749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:09.252365 8749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:09.252447 8749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:06:09.258967 8749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:35141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:09.260061 8749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:09.261364 8749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:09.271589 8754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:09.271461 8755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:09.271724 8749 server_base.cc:1034] running on GCE node
W20260924 07:06:09.271463 8757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:09.272382 8749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:09.273200 8749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:09.274384 8749 hybrid_clock.cc:648] HybridClock initialized: now 1790233569274343 us; error 51 us; skew 500 ppm
I20260924 07:06:09.276343 8749 webserver.cc:460] Webserver started at http://127.3.217.2:45061/ using document root <none> and password file <none>
I20260924 07:06:09.276891 8749 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:09.276949 8749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:09.277102 8749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:09.278990 8749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "3c042c730b7446bebd8e83e8e0871376"
format_stamp: "Formatted at 2026-09-24 07:06:09 on dist-test-slave-2p7j"
server_key: "24a2969065387366a62af7fece3a87c2"
server_key_iv: "15a7dbd27bc33fa6fd2bf7f9b97b090c"
server_key_version: "encryptionkey@0"
I20260924 07:06:09.279458 8749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "3c042c730b7446bebd8e83e8e0871376"
format_stamp: "Formatted at 2026-09-24 07:06:09 on dist-test-slave-2p7j"
server_key: "24a2969065387366a62af7fece3a87c2"
server_key_iv: "15a7dbd27bc33fa6fd2bf7f9b97b090c"
server_key_version: "encryptionkey@0"
I20260924 07:06:09.283301 8749 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.001s
I20260924 07:06:09.286039 8763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:09.287441 8749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:06:09.287575 8749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "3c042c730b7446bebd8e83e8e0871376"
format_stamp: "Formatted at 2026-09-24 07:06:09 on dist-test-slave-2p7j"
server_key: "24a2969065387366a62af7fece3a87c2"
server_key_iv: "15a7dbd27bc33fa6fd2bf7f9b97b090c"
server_key_version: "encryptionkey@0"
I20260924 07:06:09.287684 8749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:09.292104 8749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:09.292665 8749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:09.292805 8749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:09.293282 8749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:09.294128 8749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:09.294174 8749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:09.294209 8749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:09.294224 8749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:09.303773 8749 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:46723
I20260924 07:06:09.303856 8876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:46723 every 8 connection(s)
I20260924 07:06:09.304663 8749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:06:09.311383 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:09.306409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:58785 (local address 127.3.217.62:35141)
0924 07:06:09.306549 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:09.306553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:09.307985 (+ 1432us) server_negotiation.cc:407] Connection header received
0924 07:06:09.309296 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:09.309298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:09.309335 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:09.309408 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:09.310623 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:09.310628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:09.310630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:06:09.310651 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:09.310686 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:06:09.310698 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:09.310884 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:09.311168 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 07:06:09.311235 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:06:09.312044 8879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:09.306764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:35141 (local address 127.3.217.2:58785)
0924 07:06:09.307821 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:09.307866 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:09.309081 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:09.309560 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:09.309566 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:09.309650 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:09.310262 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:06:09.310264 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:09.310301 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:06:09.310369 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:09.310914 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:09.310922 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:09.310927 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:06:09.311075 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 07:06:09.311281 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":119,"threads_started":1}
I20260924 07:06:09.313575 8877 heartbeater.cc:347] Connected to a master server at 127.3.217.62:35141
I20260924 07:06:09.313690 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 8749
I20260924 07:06:09.313802 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:06:09.313831 8877 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:09.314097 3940 external_mini_cluster.cc:1428] Setting key 0e88bcba4f12594c8c00ddd4e410ade8
I20260924 07:06:09.314242 8877 heartbeater.cc:511] Master 127.3.217.62:35141 requested a full tablet report, sending...
I20260924 07:06:09.315654 8557 ts_manager.cc:194] Registered new tserver with Master: 3c042c730b7446bebd8e83e8e0871376 (127.3.217.2:46723)
I20260924 07:06:09.316242 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:35141
--builtin_ntp_servers=127.3.217.20:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 07:06:09.424999 8880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:09.425242 8880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:09.425436 8880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:09.429621 8880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:09.429689 8880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:09.429840 8880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:06:09.436010 8880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:35141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:09.437103 8880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:09.438443 8880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:09.448225 8886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:09.448206 8885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:09.448464 8888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:09.448726 8880 server_base.cc:1034] running on GCE node
I20260924 07:06:09.449157 8880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:09.449795 8880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:09.450973 8880 hybrid_clock.cc:648] HybridClock initialized: now 1790233569450942 us; error 59 us; skew 500 ppm
I20260924 07:06:09.453411 8880 webserver.cc:460] Webserver started at http://127.3.217.3:45129/ using document root <none> and password file <none>
I20260924 07:06:09.454202 8880 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:09.454279 8880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:09.454502 8880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:09.456959 8880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "f9e61462fd314f44957587767de7bb78"
format_stamp: "Formatted at 2026-09-24 07:06:09 on dist-test-slave-2p7j"
server_key: "25736279073a31127632fdf08e5a03ff"
server_key_iv: "fac1c27b730399b1c16905cc873b2cc1"
server_key_version: "encryptionkey@0"
I20260924 07:06:09.457592 8880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "f9e61462fd314f44957587767de7bb78"
format_stamp: "Formatted at 2026-09-24 07:06:09 on dist-test-slave-2p7j"
server_key: "25736279073a31127632fdf08e5a03ff"
server_key_iv: "fac1c27b730399b1c16905cc873b2cc1"
server_key_version: "encryptionkey@0"
I20260924 07:06:09.461519 8880 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:06:09.464090 8894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:09.465307 8880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:06:09.465747 8880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "f9e61462fd314f44957587767de7bb78"
format_stamp: "Formatted at 2026-09-24 07:06:09 on dist-test-slave-2p7j"
server_key: "25736279073a31127632fdf08e5a03ff"
server_key_iv: "fac1c27b730399b1c16905cc873b2cc1"
server_key_version: "encryptionkey@0"
I20260924 07:06:09.465873 8880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:09.470712 8880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:09.471287 8880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:09.471419 8880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:09.471971 8880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:09.472810 8880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:09.472858 8880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:09.472895 8880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:09.472923 8880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:09.482343 8880 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:38031
I20260924 07:06:09.482383 9007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:38031 every 8 connection(s)
I20260924 07:06:09.483140 8880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:06:09.489243 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:09.484814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:47943 (local address 127.3.217.62:35141)
0924 07:06:09.484986 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:09.484991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:09.486094 (+ 1103us) server_negotiation.cc:407] Connection header received
0924 07:06:09.487493 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:09.487496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:09.487561 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:09.487635 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:09.488789 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:09.488793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:09.488795 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:06:09.488806 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:09.488823 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:06:09.488831 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:09.488949 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:09.489060 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 07:06:09.489104 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 07:06:09.490068 9010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:09.485192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:35141 (local address 127.3.217.3:47943)
0924 07:06:09.485811 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:09.485848 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:09.487267 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:09.487798 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:09.487804 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:09.487896 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:09.488523 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:06:09.488525 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:09.488568 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:06:09.488636 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:09.488933 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:09.488941 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:09.488946 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:06:09.489095 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 07:06:09.489337 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":107,"threads_started":1}
I20260924 07:06:09.491118 9008 heartbeater.cc:347] Connected to a master server at 127.3.217.62:35141
I20260924 07:06:09.491323 9008 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:09.491758 9008 heartbeater.cc:511] Master 127.3.217.62:35141 requested a full tablet report, sending...
I20260924 07:06:09.491968 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 8880
I20260924 07:06:09.492070 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:06:09.492347 3940 external_mini_cluster.cc:1428] Setting key 0f5948532d101b385c18d7daa47029d5
I20260924 07:06:09.492846 8557 ts_manager.cc:194] Registered new tserver with Master: f9e61462fd314f44957587767de7bb78 (127.3.217.3:38031)
I20260924 07:06:09.494186 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:06:09.497228 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:09.495931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36760 (local address 127.3.217.62:35141)
0924 07:06:09.496031 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:09.496036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:09.496195 (+ 159us) server_negotiation.cc:407] Connection header received
0924 07:06:09.496274 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:09.496276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:09.496301 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:09.496363 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:09.496983 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:09.496986 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:09.496988 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:06:09.496996 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:09.497011 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:06:09.497018 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:09.497080 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:09.497118 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 07:06:09.497146 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 07:06:09.499621 8557 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:06:09.502429 8557 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:06:09.514974 9023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:09.511366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:39821 (local address 127.0.0.1:54204)
0924 07:06:09.512935 (+ 1569us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:09.512950 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:09.513046 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:09.513452 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:09.513458 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:09.513549 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:09.514054 (+ 505us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:06:09.514056 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:09.514083 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:06:09.514156 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:09.514635 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:09.514644 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:09.514650 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:06:09.514760 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 07:06:09.514824 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1483,"thread_start_us":144,"threads_started":1}
I20260924 07:06:09.515547 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:09.511314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:38031 (local address 127.0.0.1:35896)
0924 07:06:09.511788 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:09.511847 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:09.511970 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:09.513462 (+ 1492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:09.513466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:09.513551 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:09.514257 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:06:09.514259 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:09.514272 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:06:09.514303 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:09.515303 (+ 1000us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:09.515306 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:09.515309 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:06:09.515368 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 07:06:09.515402 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":118,"threads_started":1}
I20260924 07:06:09.515720 9021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:09.511407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35896 (local address 127.3.217.3:38031)
0924 07:06:09.512901 (+ 1494us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:09.512911 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:09.512937 (+ 26us) server_negotiation.cc:407] Connection header received
0924 07:06:09.512991 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:09.512996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:09.513264 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:09.513937 (+ 673us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:09.515017 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:09.515110 (+ 93us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:09.515139 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:06:09.515153 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:09.515196 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:06:09.515225 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:09.515330 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:09.515378 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 07:06:09.515456 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1363,"thread_start_us":168,"threads_started":1}
I20260924 07:06:09.515944 9022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:09.512180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54204 (local address 127.3.217.1:39821)
0924 07:06:09.512595 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:09.512825 (+ 230us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:09.513043 (+ 218us) server_negotiation.cc:407] Connection header received
0924 07:06:09.513152 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:09.513156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:09.513301 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:09.513471 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:09.514447 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:09.514453 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:09.514465 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:06:09.514478 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:09.514520 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:06:09.514546 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:09.514657 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:09.514781 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 07:06:09.515790 (+ 1009us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":119,"threads_started":1}
I20260924 07:06:09.516497 9020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:09.512566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:46723 (local address 127.0.0.1:33238)
0924 07:06:09.513218 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:09.513227 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:09.513286 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:09.514212 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:09.514215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:09.514232 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:09.514995 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:06:09.514996 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:09.515006 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:06:09.515029 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:09.515562 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:09.515565 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:09.515568 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:06:09.515715 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 07:06:09.515745 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":97,"threads_started":1}
I20260924 07:06:09.516950 9024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:09.512720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33238 (local address 127.3.217.2:46723)
0924 07:06:09.513082 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:09.513095 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:09.513302 (+ 207us) server_negotiation.cc:407] Connection header received
0924 07:06:09.513362 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:09.513367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:09.513581 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:09.513721 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:09.515309 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:09.515315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:09.515325 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:06:09.515336 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:09.515377 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:06:09.515415 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:09.515522 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:09.516733 (+ 1211us) server_negotiation.cc:299] Negotiation successful
0924 07:06:09.516795 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":85,"threads_started":1}
I20260924 07:06:09.518314 8680 tablet_service.cc:1467] Processing CreateTablet for tablet e789a526c9d04e00b26916b9d038a054 (DEFAULT_TABLE table=test-table [id=ba42d94605ff411097cdb08aec43a8bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:09.518323 8942 tablet_service.cc:1467] Processing CreateTablet for tablet e789a526c9d04e00b26916b9d038a054 (DEFAULT_TABLE table=test-table [id=ba42d94605ff411097cdb08aec43a8bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:09.519280 8680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e789a526c9d04e00b26916b9d038a054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:09.519320 8942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e789a526c9d04e00b26916b9d038a054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:09.519332 8811 tablet_service.cc:1467] Processing CreateTablet for tablet e789a526c9d04e00b26916b9d038a054 (DEFAULT_TABLE table=test-table [id=ba42d94605ff411097cdb08aec43a8bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:09.520200 8811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e789a526c9d04e00b26916b9d038a054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:09.525486 9025 tablet_bootstrap.cc:492] T e789a526c9d04e00b26916b9d038a054 P cae33adbb3344ff99b0ef44e943eaaa0: Bootstrap starting.
I20260924 07:06:09.526556 9027 tablet_bootstrap.cc:492] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376: Bootstrap starting.
I20260924 07:06:09.527997 9026 tablet_bootstrap.cc:492] T e789a526c9d04e00b26916b9d038a054 P f9e61462fd314f44957587767de7bb78: Bootstrap starting.
I20260924 07:06:09.528103 9025 tablet_bootstrap.cc:654] T e789a526c9d04e00b26916b9d038a054 P cae33adbb3344ff99b0ef44e943eaaa0: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:09.528633 9027 tablet_bootstrap.cc:654] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:09.529297 9025 log.cc:824] T e789a526c9d04e00b26916b9d038a054 P cae33adbb3344ff99b0ef44e943eaaa0: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:09.529341 9027 log.cc:824] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:09.529729 9026 tablet_bootstrap.cc:654] T e789a526c9d04e00b26916b9d038a054 P f9e61462fd314f44957587767de7bb78: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:09.530397 9026 log.cc:824] T e789a526c9d04e00b26916b9d038a054 P f9e61462fd314f44957587767de7bb78: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:09.531144 9025 tablet_bootstrap.cc:492] T e789a526c9d04e00b26916b9d038a054 P cae33adbb3344ff99b0ef44e943eaaa0: No bootstrap required, opened a new log
I20260924 07:06:09.531342 9025 ts_tablet_manager.cc:1397] T e789a526c9d04e00b26916b9d038a054 P cae33adbb3344ff99b0ef44e943eaaa0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:06:09.531740 9027 tablet_bootstrap.cc:492] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376: No bootstrap required, opened a new log
I20260924 07:06:09.531913 9027 ts_tablet_manager.cc:1397] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:06:09.531925 9026 tablet_bootstrap.cc:492] T e789a526c9d04e00b26916b9d038a054 P f9e61462fd314f44957587767de7bb78: No bootstrap required, opened a new log
I20260924 07:06:09.532078 9026 ts_tablet_manager.cc:1397] T e789a526c9d04e00b26916b9d038a054 P f9e61462fd314f44957587767de7bb78: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 07:06:09.535259 9026 raft_consensus.cc:348] T e789a526c9d04e00b26916b9d038a054 P f9e61462fd314f44957587767de7bb78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e61462fd314f44957587767de7bb78" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 38031 } } peers { permanent_uuid: "cae33adbb3344ff99b0ef44e943eaaa0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 39821 } } peers { permanent_uuid: "3c042c730b7446bebd8e83e8e0871376" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 46723 } }
I20260924 07:06:09.535466 9026 raft_consensus.cc:374] T e789a526c9d04e00b26916b9d038a054 P f9e61462fd314f44957587767de7bb78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:09.535535 9026 raft_consensus.cc:729] T e789a526c9d04e00b26916b9d038a054 P f9e61462fd314f44957587767de7bb78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e61462fd314f44957587767de7bb78, State: Initialized, Role: FOLLOWER
I20260924 07:06:09.535781 9025 raft_consensus.cc:348] T e789a526c9d04e00b26916b9d038a054 P cae33adbb3344ff99b0ef44e943eaaa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e61462fd314f44957587767de7bb78" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 38031 } } peers { permanent_uuid: "cae33adbb3344ff99b0ef44e943eaaa0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 39821 } } peers { permanent_uuid: "3c042c730b7446bebd8e83e8e0871376" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 46723 } }
I20260924 07:06:09.535923 9026 consensus_queue.cc:260] T e789a526c9d04e00b26916b9d038a054 P f9e61462fd314f44957587767de7bb78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e61462fd314f44957587767de7bb78" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 38031 } } peers { permanent_uuid: "cae33adbb3344ff99b0ef44e943eaaa0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 39821 } } peers { permanent_uuid: "3c042c730b7446bebd8e83e8e0871376" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 46723 } }
I20260924 07:06:09.536026 9025 raft_consensus.cc:374] T e789a526c9d04e00b26916b9d038a054 P cae33adbb3344ff99b0ef44e943eaaa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:09.536087 9025 raft_consensus.cc:729] T e789a526c9d04e00b26916b9d038a054 P cae33adbb3344ff99b0ef44e943eaaa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae33adbb3344ff99b0ef44e943eaaa0, State: Initialized, Role: FOLLOWER
I20260924 07:06:09.536337 9027 raft_consensus.cc:348] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e61462fd314f44957587767de7bb78" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 38031 } } peers { permanent_uuid: "cae33adbb3344ff99b0ef44e943eaaa0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 39821 } } peers { permanent_uuid: "3c042c730b7446bebd8e83e8e0871376" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 46723 } }
I20260924 07:06:09.536482 9025 consensus_queue.cc:260] T e789a526c9d04e00b26916b9d038a054 P cae33adbb3344ff99b0ef44e943eaaa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e61462fd314f44957587767de7bb78" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 38031 } } peers { permanent_uuid: "cae33adbb3344ff99b0ef44e943eaaa0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 39821 } } peers { permanent_uuid: "3c042c730b7446bebd8e83e8e0871376" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 46723 } }
I20260924 07:06:09.536588 9027 raft_consensus.cc:374] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:09.536665 9027 raft_consensus.cc:729] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c042c730b7446bebd8e83e8e0871376, State: Initialized, Role: FOLLOWER
I20260924 07:06:09.536742 9026 ts_tablet_manager.cc:1428] T e789a526c9d04e00b26916b9d038a054 P f9e61462fd314f44957587767de7bb78: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 07:06:09.537052 9008 heartbeater.cc:503] Master 127.3.217.62:35141 was elected leader, sending a full tablet report...
I20260924 07:06:09.537068 9027 consensus_queue.cc:260] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e61462fd314f44957587767de7bb78" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 38031 } } peers { permanent_uuid: "cae33adbb3344ff99b0ef44e943eaaa0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 39821 } } peers { permanent_uuid: "3c042c730b7446bebd8e83e8e0871376" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 46723 } }
I20260924 07:06:09.537422 9025 ts_tablet_manager.cc:1428] T e789a526c9d04e00b26916b9d038a054 P cae33adbb3344ff99b0ef44e943eaaa0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 07:06:09.537688 8877 heartbeater.cc:503] Master 127.3.217.62:35141 was elected leader, sending a full tablet report...
I20260924 07:06:09.537726 8746 heartbeater.cc:503] Master 127.3.217.62:35141 was elected leader, sending a full tablet report...
I20260924 07:06:09.539218 9027 ts_tablet_manager.cc:1428] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260924 07:06:09.555673 8878 tablet.cc:2369] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:06:09.626428 8747 tablet.cc:2369] T e789a526c9d04e00b26916b9d038a054 P cae33adbb3344ff99b0ef44e943eaaa0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:06:09.734302 9009 tablet.cc:2369] T e789a526c9d04e00b26916b9d038a054 P f9e61462fd314f44957587767de7bb78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:06:09.793926 9033 raft_consensus.cc:482] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:06:09.794137 9033 raft_consensus.cc:504] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e61462fd314f44957587767de7bb78" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 38031 } } peers { permanent_uuid: "cae33adbb3344ff99b0ef44e943eaaa0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 39821 } } peers { permanent_uuid: "3c042c730b7446bebd8e83e8e0871376" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 46723 } }
I20260924 07:06:09.795329 9033 leader_election.cc:288] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9e61462fd314f44957587767de7bb78 (127.3.217.3:38031), cae33adbb3344ff99b0ef44e943eaaa0 (127.3.217.1:39821)
I20260924 07:06:09.797358 8879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:09.795838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:38031 (local address 127.3.217.2:55891)
0924 07:06:09.795991 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:09.796007 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:09.796103 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:09.796340 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:09.796343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:09.796357 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:09.796814 (+ 457us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:06:09.796816 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:09.796828 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:06:09.796855 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:09.797103 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:09.797106 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:09.797109 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:06:09.797183 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 07:06:09.797217 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 07:06:09.797379 9021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:09.795966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:55891 (local address 127.3.217.3:38031)
0924 07:06:09.796130 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:09.796134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:09.796149 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:06:09.796193 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:09.796196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:09.796229 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:09.796311 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:09.796976 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:09.796980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:09.796982 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:06:09.796994 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:09.797014 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:06:09.797025 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:09.797100 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:09.797195 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 07:06:09.797236 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 07:06:09.797425 9022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:09.796069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:43835 (local address 127.3.217.1:39821)
0924 07:06:09.796191 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:09.796196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:09.796295 (+ 99us) server_negotiation.cc:407] Connection header received
0924 07:06:09.796398 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:09.796401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:09.796432 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:09.796494 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:09.797090 (+ 596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:09.797094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:09.797096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:06:09.797108 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:09.797127 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:06:09.797136 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:09.797214 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:09.797285 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 07:06:09.797318 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:06:09.797488 9034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:09.795850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:39821 (local address 127.3.217.2:43835)
0924 07:06:09.796211 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:09.796224 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:09.796309 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:09.796525 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:09.796528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:09.796542 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:09.796942 (+ 400us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:06:09.796943 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:09.796953 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:06:09.796975 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:09.797210 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:09.797213 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:09.797216 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:06:09.797287 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 07:06:09.797318 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":21,"thread_start_us":98,"threads_started":1}
I20260924 07:06:09.798074 8699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e789a526c9d04e00b26916b9d038a054" candidate_uuid: "3c042c730b7446bebd8e83e8e0871376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae33adbb3344ff99b0ef44e943eaaa0" is_pre_election: true
I20260924 07:06:09.798487 8962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e789a526c9d04e00b26916b9d038a054" candidate_uuid: "3c042c730b7446bebd8e83e8e0871376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e61462fd314f44957587767de7bb78" is_pre_election: true
I20260924 07:06:09.798457 8699 raft_consensus.cc:2454] T e789a526c9d04e00b26916b9d038a054 P cae33adbb3344ff99b0ef44e943eaaa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c042c730b7446bebd8e83e8e0871376 in term 0.
I20260924 07:06:09.798730 8962 raft_consensus.cc:2454] T e789a526c9d04e00b26916b9d038a054 P f9e61462fd314f44957587767de7bb78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c042c730b7446bebd8e83e8e0871376 in term 0.
I20260924 07:06:09.799047 8764 leader_election.cc:302] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c042c730b7446bebd8e83e8e0871376, cae33adbb3344ff99b0ef44e943eaaa0; no voters:
I20260924 07:06:09.799289 9033 raft_consensus.cc:2781] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:06:09.799649 9033 raft_consensus.cc:482] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:06:09.799728 9033 raft_consensus.cc:3037] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:09.800885 9033 raft_consensus.cc:504] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e61462fd314f44957587767de7bb78" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 38031 } } peers { permanent_uuid: "cae33adbb3344ff99b0ef44e943eaaa0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 39821 } } peers { permanent_uuid: "3c042c730b7446bebd8e83e8e0871376" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 46723 } }
I20260924 07:06:09.801179 9033 leader_election.cc:288] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376 [CANDIDATE]: Term 1 election: Requested vote from peers f9e61462fd314f44957587767de7bb78 (127.3.217.3:38031), cae33adbb3344ff99b0ef44e943eaaa0 (127.3.217.1:39821)
I20260924 07:06:09.801540 8962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e789a526c9d04e00b26916b9d038a054" candidate_uuid: "3c042c730b7446bebd8e83e8e0871376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e61462fd314f44957587767de7bb78"
I20260924 07:06:09.801671 8962 raft_consensus.cc:3037] T e789a526c9d04e00b26916b9d038a054 P f9e61462fd314f44957587767de7bb78 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:09.801851 8699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e789a526c9d04e00b26916b9d038a054" candidate_uuid: "3c042c730b7446bebd8e83e8e0871376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae33adbb3344ff99b0ef44e943eaaa0"
I20260924 07:06:09.802011 8699 raft_consensus.cc:3037] T e789a526c9d04e00b26916b9d038a054 P cae33adbb3344ff99b0ef44e943eaaa0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:09.802790 8962 raft_consensus.cc:2454] T e789a526c9d04e00b26916b9d038a054 P f9e61462fd314f44957587767de7bb78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c042c730b7446bebd8e83e8e0871376 in term 1.
I20260924 07:06:09.803115 8766 leader_election.cc:302] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c042c730b7446bebd8e83e8e0871376, f9e61462fd314f44957587767de7bb78; no voters:
I20260924 07:06:09.803263 9033 raft_consensus.cc:2781] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:09.803268 8699 raft_consensus.cc:2454] T e789a526c9d04e00b26916b9d038a054 P cae33adbb3344ff99b0ef44e943eaaa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c042c730b7446bebd8e83e8e0871376 in term 1.
I20260924 07:06:09.803489 9033 raft_consensus.cc:686] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376 [term 1 LEADER]: Becoming Leader. State: Replica: 3c042c730b7446bebd8e83e8e0871376, State: Running, Role: LEADER
I20260924 07:06:09.803833 9033 consensus_queue.cc:237] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e61462fd314f44957587767de7bb78" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 38031 } } peers { permanent_uuid: "cae33adbb3344ff99b0ef44e943eaaa0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 39821 } } peers { permanent_uuid: "3c042c730b7446bebd8e83e8e0871376" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 46723 } }
I20260924 07:06:09.806655 8558 catalog_manager.cc:5495] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c042c730b7446bebd8e83e8e0871376 (127.3.217.2). New cstate: current_term: 1 leader_uuid: "3c042c730b7446bebd8e83e8e0871376" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e61462fd314f44957587767de7bb78" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cae33adbb3344ff99b0ef44e943eaaa0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 39821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c042c730b7446bebd8e83e8e0871376" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 46723 } health_report { overall_health: HEALTHY } } }
I20260924 07:06:09.879956 9024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:09.878343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33240 (local address 127.3.217.2:46723)
0924 07:06:09.878493 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:09.878499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:09.878518 (+ 19us) server_negotiation.cc:407] Connection header received
0924 07:06:09.878720 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:09.878723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:09.878793 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:09.878920 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:09.879584 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:09.879588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:09.879590 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:06:09.879603 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:09.879622 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:06:09.879630 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:09.879702 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:09.879772 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 07:06:09.879814 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:06:09.885423 8962 raft_consensus.cc:1261] T e789a526c9d04e00b26916b9d038a054 P f9e61462fd314f44957587767de7bb78 [term 1 FOLLOWER]: Refusing update from remote peer 3c042c730b7446bebd8e83e8e0871376: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:06:09.885423 8699 raft_consensus.cc:1261] T e789a526c9d04e00b26916b9d038a054 P cae33adbb3344ff99b0ef44e943eaaa0 [term 1 FOLLOWER]: Refusing update from remote peer 3c042c730b7446bebd8e83e8e0871376: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:06:09.886138 9035 consensus_queue.cc:1035] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e61462fd314f44957587767de7bb78" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:06:09.886319 9033 consensus_queue.cc:1035] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae33adbb3344ff99b0ef44e943eaaa0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 39821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:06:09.902385 8557 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:06:09.902521 8557 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:06:09.904606 8557 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5268e16f2d624cf09304a39881db75a4: Sending DeleteTablet for 3 replicas of tablet e789a526c9d04e00b26916b9d038a054
I20260924 07:06:09.905303 8942 tablet_service.cc:1514] Processing DeleteTablet for tablet e789a526c9d04e00b26916b9d038a054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:09 UTC) from {username='slave'} at 127.0.0.1:35896
I20260924 07:06:09.905351 8680 tablet_service.cc:1514] Processing DeleteTablet for tablet e789a526c9d04e00b26916b9d038a054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:09 UTC) from {username='slave'} at 127.0.0.1:54204
I20260924 07:06:09.905483 8811 tablet_service.cc:1514] Processing DeleteTablet for tablet e789a526c9d04e00b26916b9d038a054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:09 UTC) from {username='slave'} at 127.0.0.1:33238
I20260924 07:06:09.905920 9050 tablet_replica.cc:330] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376: stopping tablet replica
I20260924 07:06:09.905956 9048 tablet_replica.cc:330] T e789a526c9d04e00b26916b9d038a054 P f9e61462fd314f44957587767de7bb78: stopping tablet replica
I20260924 07:06:09.906230 9048 raft_consensus.cc:2229] T e789a526c9d04e00b26916b9d038a054 P f9e61462fd314f44957587767de7bb78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:06:09.906272 9050 raft_consensus.cc:2229] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:06:09.906477 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 8618
I20260924 07:06:09.906504 9048 raft_consensus.cc:2258] T e789a526c9d04e00b26916b9d038a054 P f9e61462fd314f44957587767de7bb78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:09.906617 9050 raft_consensus.cc:2258] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:09.908170 9050 ts_tablet_manager.cc:1905] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:06:09.910209 9048 ts_tablet_manager.cc:1905] T e789a526c9d04e00b26916b9d038a054 P f9e61462fd314f44957587767de7bb78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:06:09.910754 9050 ts_tablet_manager.cc:1918] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:06:09.910843 9050 log.cc:1196] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/e789a526c9d04e00b26916b9d038a054
I20260924 07:06:09.911113 9050 ts_tablet_manager.cc:1939] T e789a526c9d04e00b26916b9d038a054 P 3c042c730b7446bebd8e83e8e0871376: Deleting consensus metadata
I20260924 07:06:09.912168 8545 catalog_manager.cc:4841] TS 3c042c730b7446bebd8e83e8e0871376 (127.3.217.2:46723): tablet e789a526c9d04e00b26916b9d038a054 (table test-table [id=ba42d94605ff411097cdb08aec43a8bd]) successfully deleted
I20260924 07:06:09.913139 9048 ts_tablet_manager.cc:1918] T e789a526c9d04e00b26916b9d038a054 P f9e61462fd314f44957587767de7bb78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:06:09.913232 9048 log.cc:1196] T e789a526c9d04e00b26916b9d038a054 P f9e61462fd314f44957587767de7bb78: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/e789a526c9d04e00b26916b9d038a054
I20260924 07:06:09.913499 9048 ts_tablet_manager.cc:1939] T e789a526c9d04e00b26916b9d038a054 P f9e61462fd314f44957587767de7bb78: Deleting consensus metadata
W20260924 07:06:09.914086 8542 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.217.1:39821 (error 108)
I20260924 07:06:09.914513 8544 catalog_manager.cc:4841] TS f9e61462fd314f44957587767de7bb78 (127.3.217.3:38031): tablet e789a526c9d04e00b26916b9d038a054 (table test-table [id=ba42d94605ff411097cdb08aec43a8bd]) successfully deleted
I20260924 07:06:09.914759 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 8749
I20260924 07:06:09.914968 9020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:09.914677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:39821 (local address 127.0.0.1:54208)
0924 07:06:09.914810 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:09.914880 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.1:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 07:06:09.915565 8542 catalog_manager.cc:4584] TS cae33adbb3344ff99b0ef44e943eaaa0 (127.3.217.1:39821): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e789a526c9d04e00b26916b9d038a054: Network error: Client connection negotiation failed: client connection to 127.3.217.1:39821: connect: Connection refused (error 111)
I20260924 07:06:09.921049 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 8880
I20260924 07:06:09.927495 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 8527
2026-09-24T07:06:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1265 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 07:06:09.949465 3940 mini_oidc.cc:152] Starting JWKS server
I20260924 07:06:09.950922 3940 webserver.cc:460] Webserver started at https://127.0.0.1:34443/ using document root <none> and password file <none>
I20260924 07:06:09.951056 3940 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 07:06:09.951258 3940 webserver.cc:460] Webserver started at http://127.0.0.1:33325/ using document root <none> and password file <none>
2026-09-24T07:06:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:06:09Z Disabled control of system clock
I20260924 07:06:09.959355 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:45681
--webserver_interface=127.3.217.62
--webserver_port=0
--builtin_ntp_servers=127.3.217.20:44121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:45681
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34443/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233541052846-3940-0/testchainca.cert with env {}
W20260924 07:06:10.070106 9057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:10.070358 9057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:10.070397 9057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:10.074436 9057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:06:10.074491 9057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:10.074505 9057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:10.074520 9057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:06:10.074534 9057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 07:06:10.074548 9057 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 07:06:10.074566 9057 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34443/jwks.json
I20260924 07:06:10.080720 9057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:44121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:45681
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:45681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34443/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233541052846-3940-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:10.081616 9057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:10.082504 9057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:10.091564 9063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:10.091589 9065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:10.091606 9062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:10.091642 9057 server_base.cc:1034] running on GCE node
I20260924 07:06:10.092375 9057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:10.093369 9057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:10.094527 9057 hybrid_clock.cc:648] HybridClock initialized: now 1790233570094503 us; error 35 us; skew 500 ppm
I20260924 07:06:10.096220 9057 webserver.cc:460] Webserver started at http://127.3.217.62:42233/ using document root <none> and password file <none>
I20260924 07:06:10.096740 9057 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:10.096804 9057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:10.096982 9057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:10.098866 9057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "f14ed3faa9784596b38e31ca5015c013"
format_stamp: "Formatted at 2026-09-24 07:06:10 on dist-test-slave-2p7j"
server_key: "7ffd9eb2488a41420408ac4fcf2b190d"
server_key_iv: "83780cfb029e533d3b18458805506b34"
server_key_version: "encryptionkey@0"
I20260924 07:06:10.099381 9057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "f14ed3faa9784596b38e31ca5015c013"
format_stamp: "Formatted at 2026-09-24 07:06:10 on dist-test-slave-2p7j"
server_key: "7ffd9eb2488a41420408ac4fcf2b190d"
server_key_iv: "83780cfb029e533d3b18458805506b34"
server_key_version: "encryptionkey@0"
I20260924 07:06:10.103335 9057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.001s
I20260924 07:06:10.105665 9071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:10.106878 9057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:06:10.107055 9057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "f14ed3faa9784596b38e31ca5015c013"
format_stamp: "Formatted at 2026-09-24 07:06:10 on dist-test-slave-2p7j"
server_key: "7ffd9eb2488a41420408ac4fcf2b190d"
server_key_iv: "83780cfb029e533d3b18458805506b34"
server_key_version: "encryptionkey@0"
I20260924 07:06:10.107152 9057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:10.131778 9057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:10.132417 9057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:10.132591 9057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:10.139295 9057 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:45681
I20260924 07:06:10.139474 9123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:45681 every 8 connection(s)
I20260924 07:06:10.140272 9057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:06:10.143308 9124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:10.145270 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 9057
I20260924 07:06:10.145463 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:06:10.145886 3940 external_mini_cluster.cc:1428] Setting key 55d7b49862a06b682e228665e5013327
I20260924 07:06:10.148887 9124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f14ed3faa9784596b38e31ca5015c013: Bootstrap starting.
I20260924 07:06:10.150990 9124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f14ed3faa9784596b38e31ca5015c013: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:10.151707 9124 log.cc:824] T 00000000000000000000000000000000 P f14ed3faa9784596b38e31ca5015c013: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:10.153566 9124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f14ed3faa9784596b38e31ca5015c013: No bootstrap required, opened a new log
I20260924 07:06:10.154680 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:10.147211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52796 (local address 127.3.217.62:45681)
0924 07:06:10.147883 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:10.147898 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:10.147932 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:06:10.148713 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:10.148731 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:10.149145 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:10.149434 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:10.149827 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:10.150755 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:10.151639 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:10.152163 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:10.152323 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:10.153056 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:10.153060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:10.153071 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:06:10.153422 (+ 351us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:10.153475 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:06:10.153494 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:10.153566 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:10.153724 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 07:06:10.153979 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":129,"threads_started":1}
I20260924 07:06:10.157232 9124 raft_consensus.cc:348] T 00000000000000000000000000000000 P f14ed3faa9784596b38e31ca5015c013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f14ed3faa9784596b38e31ca5015c013" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45681 } }
I20260924 07:06:10.157478 9124 raft_consensus.cc:374] T 00000000000000000000000000000000 P f14ed3faa9784596b38e31ca5015c013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:10.157532 9124 raft_consensus.cc:729] T 00000000000000000000000000000000 P f14ed3faa9784596b38e31ca5015c013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f14ed3faa9784596b38e31ca5015c013, State: Initialized, Role: FOLLOWER
I20260924 07:06:10.157717 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:06:10.157948 9124 consensus_queue.cc:260] T 00000000000000000000000000000000 P f14ed3faa9784596b38e31ca5015c013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f14ed3faa9784596b38e31ca5015c013" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45681 } }
I20260924 07:06:10.158092 9124 raft_consensus.cc:388] T 00000000000000000000000000000000 P f14ed3faa9784596b38e31ca5015c013 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:06:10.158149 9124 raft_consensus.cc:482] T 00000000000000000000000000000000 P f14ed3faa9784596b38e31ca5015c013 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:06:10.158238 9124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f14ed3faa9784596b38e31ca5015c013 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:10.159132 9124 raft_consensus.cc:504] T 00000000000000000000000000000000 P f14ed3faa9784596b38e31ca5015c013 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f14ed3faa9784596b38e31ca5015c013" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45681 } }
I20260924 07:06:10.159431 9124 leader_election.cc:302] T 00000000000000000000000000000000 P f14ed3faa9784596b38e31ca5015c013 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f14ed3faa9784596b38e31ca5015c013; no voters:
I20260924 07:06:10.159688 9124 leader_election.cc:288] T 00000000000000000000000000000000 P f14ed3faa9784596b38e31ca5015c013 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:06:10.159873 9129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f14ed3faa9784596b38e31ca5015c013 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:10.160096 9129 raft_consensus.cc:686] T 00000000000000000000000000000000 P f14ed3faa9784596b38e31ca5015c013 [term 1 LEADER]: Becoming Leader. State: Replica: f14ed3faa9784596b38e31ca5015c013, State: Running, Role: LEADER
I20260924 07:06:10.160379 9129 consensus_queue.cc:237] T 00000000000000000000000000000000 P f14ed3faa9784596b38e31ca5015c013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f14ed3faa9784596b38e31ca5015c013" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45681 } }
I20260924 07:06:10.160645 9124 sys_catalog.cc:564] T 00000000000000000000000000000000 P f14ed3faa9784596b38e31ca5015c013 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:06:10.161749 9130 sys_catalog.cc:455] T 00000000000000000000000000000000 P f14ed3faa9784596b38e31ca5015c013 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f14ed3faa9784596b38e31ca5015c013" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f14ed3faa9784596b38e31ca5015c013" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45681 } } }
I20260924 07:06:10.161876 9130 sys_catalog.cc:458] T 00000000000000000000000000000000 P f14ed3faa9784596b38e31ca5015c013 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:10.162132 9131 sys_catalog.cc:455] T 00000000000000000000000000000000 P f14ed3faa9784596b38e31ca5015c013 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f14ed3faa9784596b38e31ca5015c013. Latest consensus state: current_term: 1 leader_uuid: "f14ed3faa9784596b38e31ca5015c013" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f14ed3faa9784596b38e31ca5015c013" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45681 } } }
I20260924 07:06:10.162245 9131 sys_catalog.cc:458] T 00000000000000000000000000000000 P f14ed3faa9784596b38e31ca5015c013 [sys.catalog]: This master's current role is: LEADER
W20260924 07:06:10.164852 9144 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f14ed3faa9784596b38e31ca5015c013: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:06:10.164925 9144 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:06:10.164992 9138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:06:10.165695 9138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:06:10.170123 9138 catalog_manager.cc:1347] Generated new cluster ID: 88731a89d5384508875dc95ff9328f89
I20260924 07:06:10.170179 9138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:06:10.195271 9138 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:06:10.196496 9138 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:06:10.201319 9138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f14ed3faa9784596b38e31ca5015c013: Generated new TSK 0
I20260924 07:06:10.201908 9138 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 07:06:10.235400 9154 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 07:06:10.235549 9154 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 07:06:10.236029 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:10.216006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52806 (local address 127.3.217.62:45681)
0924 07:06:10.216118 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:10.216122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:10.216304 (+ 182us) server_negotiation.cc:407] Connection header received
0924 07:06:10.216535 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:10.216538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:10.216583 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:10.216664 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0924 07:06:10.216902 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:10.217422 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:10.218684 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:10.218934 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:10.219039 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:10.219557 (+ 518us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 07:06:10.235704 (+ 16147us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 07:06:10.235918 (+ 214us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52806: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 07:06:10.236099 9127 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52806: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 07:06:10.237385 9153 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:10.215918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:45681 (local address 127.0.0.1:52806)
0924 07:06:10.216231 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:10.216245 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:10.216372 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:10.216685 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:10.216687 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:10.216695 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 07:06:10.216819 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:10.216825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:10.217513 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:10.217516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:10.218535 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:10.218544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:10.219051 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:10.219055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:10.219143 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:10.219189 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 07:06:10.235911 (+ 16722us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 07:06:10.236132 (+ 221us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.217.62:45681: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":114,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.217.62:45681: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 07:06:10.238269 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 9057
2026-09-24T07:06:10Z chronyd exiting
I20260924 07:06:10.275334 3940 test_util.cc:175] -----------------------------------------------
I20260924 07:06:10.275424 3940 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233541052846-3940-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (327 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 07:06:10.276346 3940 mini_oidc.cc:152] Starting JWKS server
I20260924 07:06:10.277133 3940 webserver.cc:460] Webserver started at https://127.0.0.1:35467/ using document root <none> and password file <none>
I20260924 07:06:10.277220 3940 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 07:06:10.277438 3940 webserver.cc:460] Webserver started at http://127.0.0.1:38851/ using document root <none> and password file <none>
2026-09-24T07:06:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:06:10Z Disabled control of system clock
I20260924 07:06:10.286662 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:33985
--webserver_interface=127.3.217.62
--webserver_port=0
--builtin_ntp_servers=127.3.217.20:45529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:33985
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35467/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233541052846-3940-0/testchainca.cert with env {}
W20260924 07:06:10.395392 9160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:10.395658 9160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:10.395715 9160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:10.400192 9160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:06:10.400267 9160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:10.400291 9160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:10.400321 9160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:06:10.400350 9160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 07:06:10.400380 9160 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 07:06:10.400400 9160 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35467/jwks.json
I20260924 07:06:10.406145 9160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:45529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:33985
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:33985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35467/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233541052846-3940-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:10.407090 9160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:10.408066 9160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:10.415752 9168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:10.415750 9165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:10.415802 9166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:10.415978 9160 server_base.cc:1034] running on GCE node
I20260924 07:06:10.416522 9160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:10.417454 9160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:10.418627 9160 hybrid_clock.cc:648] HybridClock initialized: now 1790233570418609 us; error 31 us; skew 500 ppm
I20260924 07:06:10.420339 9160 webserver.cc:460] Webserver started at http://127.3.217.62:46739/ using document root <none> and password file <none>
I20260924 07:06:10.420921 9160 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:10.420987 9160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:10.421157 9160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:10.423010 9160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "d51ae0c431f040dc8721d158011cb783"
format_stamp: "Formatted at 2026-09-24 07:06:10 on dist-test-slave-2p7j"
server_key: "aea2d5f40933d0ab421f14b8dc49890b"
server_key_iv: "328c64ccca65c79bc2e729b2378ee73f"
server_key_version: "encryptionkey@0"
I20260924 07:06:10.423552 9160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "d51ae0c431f040dc8721d158011cb783"
format_stamp: "Formatted at 2026-09-24 07:06:10 on dist-test-slave-2p7j"
server_key: "aea2d5f40933d0ab421f14b8dc49890b"
server_key_iv: "328c64ccca65c79bc2e729b2378ee73f"
server_key_version: "encryptionkey@0"
I20260924 07:06:10.426719 9160 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260924 07:06:10.428952 9174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:10.430337 9160 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:06:10.430449 9160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "d51ae0c431f040dc8721d158011cb783"
format_stamp: "Formatted at 2026-09-24 07:06:10 on dist-test-slave-2p7j"
server_key: "aea2d5f40933d0ab421f14b8dc49890b"
server_key_iv: "328c64ccca65c79bc2e729b2378ee73f"
server_key_version: "encryptionkey@0"
I20260924 07:06:10.430537 9160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:10.458302 9160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:10.458935 9160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:10.459101 9160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:10.465255 9160 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:33985
I20260924 07:06:10.465278 9226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:33985 every 8 connection(s)
I20260924 07:06:10.466107 9160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:06:10.469699 9227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:10.471733 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 9160
I20260924 07:06:10.471839 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:06:10.472110 3940 external_mini_cluster.cc:1428] Setting key 8488ffde2319fa8168353e92f663a321
I20260924 07:06:10.475422 9227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d51ae0c431f040dc8721d158011cb783: Bootstrap starting.
I20260924 07:06:10.477499 9227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d51ae0c431f040dc8721d158011cb783: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:10.478354 9227 log.cc:824] T 00000000000000000000000000000000 P d51ae0c431f040dc8721d158011cb783: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:10.480201 9227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d51ae0c431f040dc8721d158011cb783: No bootstrap required, opened a new log
I20260924 07:06:10.481218 9230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:10.473415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54122 (local address 127.3.217.62:33985)
0924 07:06:10.474071 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:10.474088 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:10.474129 (+ 41us) server_negotiation.cc:407] Connection header received
0924 07:06:10.474802 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:10.474819 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:10.475177 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:10.475486 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:10.475912 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:10.476847 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:10.477979 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:10.478722 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:10.478873 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:10.479700 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:10.479705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:10.479719 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:06:10.480072 (+ 353us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:10.480121 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:06:10.480141 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:10.480223 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:10.480355 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 07:06:10.480597 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":113,"threads_started":1}
I20260924 07:06:10.483111 9227 raft_consensus.cc:348] T 00000000000000000000000000000000 P d51ae0c431f040dc8721d158011cb783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d51ae0c431f040dc8721d158011cb783" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33985 } }
I20260924 07:06:10.483338 9227 raft_consensus.cc:374] T 00000000000000000000000000000000 P d51ae0c431f040dc8721d158011cb783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:10.483402 9227 raft_consensus.cc:729] T 00000000000000000000000000000000 P d51ae0c431f040dc8721d158011cb783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d51ae0c431f040dc8721d158011cb783, State: Initialized, Role: FOLLOWER
I20260924 07:06:10.483796 9227 consensus_queue.cc:260] T 00000000000000000000000000000000 P d51ae0c431f040dc8721d158011cb783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d51ae0c431f040dc8721d158011cb783" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33985 } }
I20260924 07:06:10.483945 9227 raft_consensus.cc:388] T 00000000000000000000000000000000 P d51ae0c431f040dc8721d158011cb783 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:06:10.484006 9227 raft_consensus.cc:482] T 00000000000000000000000000000000 P d51ae0c431f040dc8721d158011cb783 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:06:10.484006 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:06:10.484097 9227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d51ae0c431f040dc8721d158011cb783 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:10.485059 9227 raft_consensus.cc:504] T 00000000000000000000000000000000 P d51ae0c431f040dc8721d158011cb783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d51ae0c431f040dc8721d158011cb783" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33985 } }
I20260924 07:06:10.485360 9227 leader_election.cc:302] T 00000000000000000000000000000000 P d51ae0c431f040dc8721d158011cb783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d51ae0c431f040dc8721d158011cb783; no voters:
I20260924 07:06:10.485646 9227 leader_election.cc:288] T 00000000000000000000000000000000 P d51ae0c431f040dc8721d158011cb783 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:06:10.485735 9232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d51ae0c431f040dc8721d158011cb783 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:10.485939 9232 raft_consensus.cc:686] T 00000000000000000000000000000000 P d51ae0c431f040dc8721d158011cb783 [term 1 LEADER]: Becoming Leader. State: Replica: d51ae0c431f040dc8721d158011cb783, State: Running, Role: LEADER
I20260924 07:06:10.486219 9232 consensus_queue.cc:237] T 00000000000000000000000000000000 P d51ae0c431f040dc8721d158011cb783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d51ae0c431f040dc8721d158011cb783" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33985 } }
I20260924 07:06:10.486748 9227 sys_catalog.cc:564] T 00000000000000000000000000000000 P d51ae0c431f040dc8721d158011cb783 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:06:10.487648 9234 sys_catalog.cc:455] T 00000000000000000000000000000000 P d51ae0c431f040dc8721d158011cb783 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d51ae0c431f040dc8721d158011cb783. Latest consensus state: current_term: 1 leader_uuid: "d51ae0c431f040dc8721d158011cb783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d51ae0c431f040dc8721d158011cb783" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33985 } } }
I20260924 07:06:10.487766 9234 sys_catalog.cc:458] T 00000000000000000000000000000000 P d51ae0c431f040dc8721d158011cb783 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:10.488183 9233 sys_catalog.cc:455] T 00000000000000000000000000000000 P d51ae0c431f040dc8721d158011cb783 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d51ae0c431f040dc8721d158011cb783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d51ae0c431f040dc8721d158011cb783" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33985 } } }
I20260924 07:06:10.488296 9241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:06:10.488313 9233 sys_catalog.cc:458] T 00000000000000000000000000000000 P d51ae0c431f040dc8721d158011cb783 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:10.491917 9241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:06:10.497293 9241 catalog_manager.cc:1347] Generated new cluster ID: 80db29da017b496783cf4f07b1b6fd59
I20260924 07:06:10.497357 9241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:06:10.531863 9241 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:06:10.533115 9241 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:06:10.540858 9241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d51ae0c431f040dc8721d158011cb783: Generated new TSK 0
I20260924 07:06:10.541540 9241 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 07:06:10.571681 9257 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 07:06:10.571766 9257 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 07:06:10.572288 9230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:10.556790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54136 (local address 127.3.217.62:33985)
0924 07:06:10.556920 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:10.556925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:10.557098 (+ 173us) server_negotiation.cc:407] Connection header received
0924 07:06:10.557221 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:10.557224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:10.557268 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:10.557354 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0924 07:06:10.557584 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:10.558131 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:10.559335 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:10.559606 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:10.559688 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:10.560206 (+ 518us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 07:06:10.571996 (+ 11790us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35467/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 07:06:10.572186 (+ 190us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54136: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35467/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 07:06:10.572338 9230 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54136: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35467/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 07:06:10.572494 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:10.556728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:33985 (local address 127.0.0.1:54136)
0924 07:06:10.556989 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:10.557003 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:10.557113 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:10.557365 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:10.557368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:10.557376 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 07:06:10.557489 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:10.557496 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:10.558234 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:10.558237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:10.559205 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:10.559214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:10.559710 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:10.559714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:10.559801 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:10.559828 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 07:06:10.572154 (+ 12326us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35467/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 07:06:10.572318 (+ 164us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.217.62:33985: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35467/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":83,"threads_started":1}
I20260924 07:06:10.573119 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 9160
2026-09-24T07:06:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (336 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 07:06:10.612820 3940 mini_oidc.cc:152] Starting JWKS server
I20260924 07:06:10.613617 3940 webserver.cc:460] Webserver started at https://127.0.0.1:41093/ using document root <none> and password file <none>
I20260924 07:06:10.613695 3940 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 07:06:10.613925 3940 webserver.cc:460] Webserver started at http://127.0.0.1:38235/ using document root <none> and password file <none>
2026-09-24T07:06:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:06:10Z Disabled control of system clock
I20260924 07:06:10.621567 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:46337
--webserver_interface=127.3.217.62
--webserver_port=0
--builtin_ntp_servers=127.3.217.20:37241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:46337
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41093/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 07:06:10.729921 9263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:10.730154 9263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:10.730206 9263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:10.734115 9263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:06:10.734169 9263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:10.734189 9263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:10.734212 9263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:06:10.734239 9263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 07:06:10.734267 9263 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 07:06:10.734292 9263 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41093/jwks.json
I20260924 07:06:10.738899 9263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:37241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:46337
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:46337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41093/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:10.739876 9263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:10.741331 9263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:10.750123 9269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:10.750144 9268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:10.750144 9271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:10.750339 9263 server_base.cc:1034] running on GCE node
I20260924 07:06:10.750998 9263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:10.752084 9263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:10.753239 9263 hybrid_clock.cc:648] HybridClock initialized: now 1790233570753225 us; error 32 us; skew 500 ppm
I20260924 07:06:10.754885 9263 webserver.cc:460] Webserver started at http://127.3.217.62:44549/ using document root <none> and password file <none>
I20260924 07:06:10.755405 9263 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:10.755486 9263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:10.755700 9263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:10.757531 9263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "d32761bd5dae404ba013db56488b1595"
format_stamp: "Formatted at 2026-09-24 07:06:10 on dist-test-slave-2p7j"
server_key: "ec0ec4b1385014a125f8f7ba926bb1fe"
server_key_iv: "bb39ce6c35b7e0b681abcd4dff051a0f"
server_key_version: "encryptionkey@0"
I20260924 07:06:10.758021 9263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "d32761bd5dae404ba013db56488b1595"
format_stamp: "Formatted at 2026-09-24 07:06:10 on dist-test-slave-2p7j"
server_key: "ec0ec4b1385014a125f8f7ba926bb1fe"
server_key_iv: "bb39ce6c35b7e0b681abcd4dff051a0f"
server_key_version: "encryptionkey@0"
I20260924 07:06:10.761739 9263 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:06:10.764014 9277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:10.765125 9263 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:06:10.765316 9263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "d32761bd5dae404ba013db56488b1595"
format_stamp: "Formatted at 2026-09-24 07:06:10 on dist-test-slave-2p7j"
server_key: "ec0ec4b1385014a125f8f7ba926bb1fe"
server_key_iv: "bb39ce6c35b7e0b681abcd4dff051a0f"
server_key_version: "encryptionkey@0"
I20260924 07:06:10.765425 9263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:10.792451 9263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:10.793062 9263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:10.793215 9263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:10.799471 9263 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:46337
I20260924 07:06:10.799571 9329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:46337 every 8 connection(s)
I20260924 07:06:10.800406 9263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:06:10.803390 9330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:10.806679 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 9263
I20260924 07:06:10.806787 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:06:10.807011 3940 external_mini_cluster.cc:1428] Setting key c624ee9b127a3e8b0fd2dd90b8419bd4
I20260924 07:06:10.808851 9330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d32761bd5dae404ba013db56488b1595: Bootstrap starting.
I20260924 07:06:10.810896 9330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d32761bd5dae404ba013db56488b1595: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:10.811576 9330 log.cc:824] T 00000000000000000000000000000000 P d32761bd5dae404ba013db56488b1595: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:10.813205 9330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d32761bd5dae404ba013db56488b1595: No bootstrap required, opened a new log
I20260924 07:06:10.816352 9333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:10.808554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39382 (local address 127.3.217.62:46337)
0924 07:06:10.809201 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:10.809216 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:10.809251 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:06:10.809895 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:10.809911 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:10.810266 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:10.810550 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:10.810957 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:10.811984 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:10.812812 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:10.813696 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:10.813895 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:10.814736 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:10.814741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:10.814751 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:06:10.815122 (+ 371us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:10.815172 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:06:10.815192 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:10.815275 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:10.815417 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 07:06:10.815665 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":129,"threads_started":1}
I20260924 07:06:10.816619 9330 raft_consensus.cc:348] T 00000000000000000000000000000000 P d32761bd5dae404ba013db56488b1595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d32761bd5dae404ba013db56488b1595" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 46337 } }
I20260924 07:06:10.816818 9330 raft_consensus.cc:374] T 00000000000000000000000000000000 P d32761bd5dae404ba013db56488b1595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:10.816872 9330 raft_consensus.cc:729] T 00000000000000000000000000000000 P d32761bd5dae404ba013db56488b1595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d32761bd5dae404ba013db56488b1595, State: Initialized, Role: FOLLOWER
I20260924 07:06:10.817229 9330 consensus_queue.cc:260] T 00000000000000000000000000000000 P d32761bd5dae404ba013db56488b1595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d32761bd5dae404ba013db56488b1595" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 46337 } }
I20260924 07:06:10.817348 9330 raft_consensus.cc:388] T 00000000000000000000000000000000 P d32761bd5dae404ba013db56488b1595 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:06:10.817408 9330 raft_consensus.cc:482] T 00000000000000000000000000000000 P d32761bd5dae404ba013db56488b1595 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:06:10.817499 9330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d32761bd5dae404ba013db56488b1595 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:10.818302 9330 raft_consensus.cc:504] T 00000000000000000000000000000000 P d32761bd5dae404ba013db56488b1595 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d32761bd5dae404ba013db56488b1595" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 46337 } }
I20260924 07:06:10.818604 9330 leader_election.cc:302] T 00000000000000000000000000000000 P d32761bd5dae404ba013db56488b1595 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d32761bd5dae404ba013db56488b1595; no voters:
I20260924 07:06:10.818861 9330 leader_election.cc:288] T 00000000000000000000000000000000 P d32761bd5dae404ba013db56488b1595 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:06:10.818930 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:06:10.818953 9335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d32761bd5dae404ba013db56488b1595 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:10.819123 9335 raft_consensus.cc:686] T 00000000000000000000000000000000 P d32761bd5dae404ba013db56488b1595 [term 1 LEADER]: Becoming Leader. State: Replica: d32761bd5dae404ba013db56488b1595, State: Running, Role: LEADER
I20260924 07:06:10.819386 9335 consensus_queue.cc:237] T 00000000000000000000000000000000 P d32761bd5dae404ba013db56488b1595 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d32761bd5dae404ba013db56488b1595" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 46337 } }
I20260924 07:06:10.819965 9330 sys_catalog.cc:564] T 00000000000000000000000000000000 P d32761bd5dae404ba013db56488b1595 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:06:10.821257 9336 sys_catalog.cc:455] T 00000000000000000000000000000000 P d32761bd5dae404ba013db56488b1595 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d32761bd5dae404ba013db56488b1595" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d32761bd5dae404ba013db56488b1595" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 46337 } } }
I20260924 07:06:10.821384 9336 sys_catalog.cc:458] T 00000000000000000000000000000000 P d32761bd5dae404ba013db56488b1595 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:10.821624 9336 sys_catalog.cc:455] T 00000000000000000000000000000000 P d32761bd5dae404ba013db56488b1595 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d32761bd5dae404ba013db56488b1595. Latest consensus state: current_term: 1 leader_uuid: "d32761bd5dae404ba013db56488b1595" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d32761bd5dae404ba013db56488b1595" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 46337 } } }
I20260924 07:06:10.821744 9336 sys_catalog.cc:458] T 00000000000000000000000000000000 P d32761bd5dae404ba013db56488b1595 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:10.821946 9348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:06:10.824473 9348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:06:10.829514 9348 catalog_manager.cc:1347] Generated new cluster ID: 36cd778a4b694fa2ba771bc1f70f36dd
I20260924 07:06:10.829581 9348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:06:10.844663 9348 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:06:10.845638 9348 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:06:10.853009 9348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d32761bd5dae404ba013db56488b1595: Generated new TSK 0
I20260924 07:06:10.853628 9348 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:06:10.868448 9333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:10.862023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39390 (local address 127.3.217.62:46337)
0924 07:06:10.862148 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:10.862153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:10.862350 (+ 197us) server_negotiation.cc:407] Connection header received
0924 07:06:10.862465 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:10.862468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:10.862513 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:10.862594 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0924 07:06:10.862828 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:10.863341 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:10.865135 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:10.865376 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:10.865462 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:10.865918 (+ 456us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 07:06:10.868077 (+ 2159us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41093/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 07:06:10.868296 (+ 219us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39390: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41093/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 07:06:10.868517 9333 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39390: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41093/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 07:06:10.870290 9359 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:10.861932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:46337 (local address 127.0.0.1:39390)
0924 07:06:10.862205 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:10.862219 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:10.862344 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:10.862604 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:10.862607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:10.862614 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0924 07:06:10.862717 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:10.862723 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:10.863607 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:10.863613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:10.864998 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:10.865011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:10.865490 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:10.865494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:10.865584 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:10.865613 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 07:06:10.868261 (+ 2648us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41093/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 07:06:10.870113 (+ 1852us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.217.62:46337: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41093/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":1647,"thread_start_us":93,"threads_started":1}
I20260924 07:06:10.870740 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 9263
E20260924 07:06:10.881611 9360 webserver.cc:533] Webserver: sslize failed:
E20260924 07:06:10.881695 9360 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T07:06:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (297 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T07:06:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:06:10Z Disabled control of system clock
I20260924 07:06:10.917774 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:45913
--webserver_interface=127.3.217.62
--webserver_port=0
--builtin_ntp_servers=127.3.217.20:46159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:45913
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233541052846-3940-0/testchainca.cert with env {}
W20260924 07:06:11.034523 9364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:11.034754 9364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:11.034790 9364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:11.038710 9364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:06:11.038763 9364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:11.038777 9364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:11.038791 9364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:06:11.038805 9364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 07:06:11.038841 9364 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 07:06:11.038864 9364 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 07:06:11.045027 9364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:46159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:45913
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:45913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233541052846-3940-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:11.045984 9364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:11.046906 9364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:11.056427 9372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:11.056447 9369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:11.056427 9370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:11.057075 9364 server_base.cc:1034] running on GCE node
I20260924 07:06:11.057660 9364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:11.058737 9364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:11.059924 9364 hybrid_clock.cc:648] HybridClock initialized: now 1790233571059906 us; error 45 us; skew 500 ppm
I20260924 07:06:11.062825 9364 webserver.cc:460] Webserver started at http://127.3.217.62:34879/ using document root <none> and password file <none>
I20260924 07:06:11.063427 9364 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:11.063551 9364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:11.063750 9364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:11.065701 9364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "ab418b9160544977a692c8971a370c7a"
format_stamp: "Formatted at 2026-09-24 07:06:11 on dist-test-slave-2p7j"
server_key: "fb2918feac83f4a45c026037e7a2af0e"
server_key_iv: "12d1682bf760dbe425e70bb64b303dbd"
server_key_version: "encryptionkey@0"
I20260924 07:06:11.066219 9364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "ab418b9160544977a692c8971a370c7a"
format_stamp: "Formatted at 2026-09-24 07:06:11 on dist-test-slave-2p7j"
server_key: "fb2918feac83f4a45c026037e7a2af0e"
server_key_iv: "12d1682bf760dbe425e70bb64b303dbd"
server_key_version: "encryptionkey@0"
I20260924 07:06:11.070238 9364 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260924 07:06:11.072845 9378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:11.074118 9364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:06:11.074237 9364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "ab418b9160544977a692c8971a370c7a"
format_stamp: "Formatted at 2026-09-24 07:06:11 on dist-test-slave-2p7j"
server_key: "fb2918feac83f4a45c026037e7a2af0e"
server_key_iv: "12d1682bf760dbe425e70bb64b303dbd"
server_key_version: "encryptionkey@0"
I20260924 07:06:11.074333 9364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:11.090068 9364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:11.091033 9364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:11.091343 9364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:11.099849 9364 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:45913
I20260924 07:06:11.099931 9430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:45913 every 8 connection(s)
I20260924 07:06:11.100814 9364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:06:11.104262 9431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:11.104400 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 9364
I20260924 07:06:11.104514 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:06:11.104803 3940 external_mini_cluster.cc:1428] Setting key d10332d486a9de8e76284a1dcd888524
I20260924 07:06:11.110543 9431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab418b9160544977a692c8971a370c7a: Bootstrap starting.
I20260924 07:06:11.112617 9431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab418b9160544977a692c8971a370c7a: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:11.113333 9431 log.cc:824] T 00000000000000000000000000000000 P ab418b9160544977a692c8971a370c7a: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:11.113940 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:11.106140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46418 (local address 127.3.217.62:45913)
0924 07:06:11.106743 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:11.106756 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:11.106794 (+ 38us) server_negotiation.cc:407] Connection header received
0924 07:06:11.107532 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:11.107552 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:11.107922 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:11.108216 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:11.108619 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:11.109547 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:11.110386 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:11.110887 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:11.111023 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:11.112132 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:11.112137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:11.112148 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:06:11.112511 (+ 363us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:11.112564 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:06:11.112582 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:11.112667 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:11.113035 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 07:06:11.113232 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":118,"threads_started":1}
I20260924 07:06:11.115212 9431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab418b9160544977a692c8971a370c7a: No bootstrap required, opened a new log
I20260924 07:06:11.116454 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:06:11.118960 9431 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab418b9160544977a692c8971a370c7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab418b9160544977a692c8971a370c7a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45913 } }
I20260924 07:06:11.119164 9431 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab418b9160544977a692c8971a370c7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:11.119220 9431 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab418b9160544977a692c8971a370c7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab418b9160544977a692c8971a370c7a, State: Initialized, Role: FOLLOWER
I20260924 07:06:11.119650 9431 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab418b9160544977a692c8971a370c7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab418b9160544977a692c8971a370c7a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45913 } }
I20260924 07:06:11.119781 9431 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab418b9160544977a692c8971a370c7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:06:11.119843 9431 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab418b9160544977a692c8971a370c7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:06:11.119940 9431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab418b9160544977a692c8971a370c7a [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:11.120857 9431 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab418b9160544977a692c8971a370c7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab418b9160544977a692c8971a370c7a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45913 } }
I20260924 07:06:11.121150 9431 leader_election.cc:302] T 00000000000000000000000000000000 P ab418b9160544977a692c8971a370c7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab418b9160544977a692c8971a370c7a; no voters:
I20260924 07:06:11.121402 9431 leader_election.cc:288] T 00000000000000000000000000000000 P ab418b9160544977a692c8971a370c7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:06:11.121483 9436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab418b9160544977a692c8971a370c7a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:11.121690 9436 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab418b9160544977a692c8971a370c7a [term 1 LEADER]: Becoming Leader. State: Replica: ab418b9160544977a692c8971a370c7a, State: Running, Role: LEADER
I20260924 07:06:11.121969 9436 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab418b9160544977a692c8971a370c7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab418b9160544977a692c8971a370c7a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45913 } }
I20260924 07:06:11.122479 9431 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab418b9160544977a692c8971a370c7a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:06:11.123582 9438 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab418b9160544977a692c8971a370c7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab418b9160544977a692c8971a370c7a. Latest consensus state: current_term: 1 leader_uuid: "ab418b9160544977a692c8971a370c7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab418b9160544977a692c8971a370c7a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45913 } } }
I20260924 07:06:11.123703 9438 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab418b9160544977a692c8971a370c7a [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:11.123991 9437 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab418b9160544977a692c8971a370c7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab418b9160544977a692c8971a370c7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab418b9160544977a692c8971a370c7a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45913 } } }
I20260924 07:06:11.124053 9445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:06:11.124148 9437 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab418b9160544977a692c8971a370c7a [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:11.126543 9445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:06:11.131958 9445 catalog_manager.cc:1347] Generated new cluster ID: 4ae96d391cc044269fa54e5257dc5803
I20260924 07:06:11.132020 9445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:06:11.144007 9445 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:06:11.144845 9445 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:06:11.152305 9445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab418b9160544977a692c8971a370c7a: Generated new TSK 0
I20260924 07:06:11.152881 9445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:06:11.289384 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:11.163061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.3.217.62:45913)
0924 07:06:11.163169 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:11.163174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:11.163347 (+ 173us) server_negotiation.cc:407] Connection header received
0924 07:06:11.163624 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:11.163627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:11.163676 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:11.163771 (+ 95us) server_negotiation.cc:226] Negotiated authn=JWT
0924 07:06:11.163995 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:11.164555 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:11.165384 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:11.165613 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:11.165697 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:11.165983 (+ 286us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 07:06:11.288923 (+122940us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 07:06:11.289219 (+ 296us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46432: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":32}
W20260924 07:06:11.289479 9434 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46432: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 07:06:11.289510 9460 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:11.162873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:45913 (local address 127.0.0.1:46432)
0924 07:06:11.163247 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:11.163268 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:11.163379 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:11.163779 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:11.163782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:11.163790 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 07:06:11.163891 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:11.163898 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:11.164659 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:11.164663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:11.165279 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:11.165285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:11.165715 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:11.165719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:11.165808 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:11.165829 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 07:06:11.289153 (+123324us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 07:06:11.289334 (+ 181us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.217.62:45913: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":107,"threads_started":1}
I20260924 07:06:11.290095 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 9364
2026-09-24T07:06:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (398 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (116 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (118 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:06:11 dist-test-slave-2p7j krb5kdc[9472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:06:11 dist-test-slave-2p7j krb5kdc[9472](info): set up 2 sockets
Sep 24 07:06:11 dist-test-slave-2p7j krb5kdc[9472](info): commencing operation
krb5kdc: starting...
W20260924 07:06:12.495939 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:06:12 dist-test-slave-2p7j krb5kdc[9472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:06:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:06:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:06:12.614511 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:37431
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:33051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:37431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:12.719440 9485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:12.719748 9485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:12.719789 9485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:12.724052 9485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:06:12.724112 9485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:12.724133 9485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:12.724148 9485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:06:12.724160 9485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:06:12.729352 9485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:33051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:37431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:37431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:12.730330 9485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:12.731262 9485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:12.739969 9491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:12.739991 9490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:12.739990 9493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:12.740401 9485 server_base.cc:1034] running on GCE node
I20260924 07:06:12.740854 9485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:12.741813 9485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:12.742986 9485 hybrid_clock.cc:648] HybridClock initialized: now 1790233572742967 us; error 30 us; skew 500 ppm
Sep 24 07:06:12 dist-test-slave-2p7j krb5kdc[9472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233572, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:12.750650 9485 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:06:12.751528 9485 webserver.cc:460] Webserver started at http://127.3.217.62:38307/ using document root <none> and password file <none>
I20260924 07:06:12.752038 9485 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:12.752094 9485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:12.752246 9485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:12.754146 9485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "5e6e2025d7fb45d39df61ae084707a1c"
format_stamp: "Formatted at 2026-09-24 07:06:12 on dist-test-slave-2p7j"
server_key: "6d5f23655bafea01dead494f5fe533a0"
server_key_iv: "23fce1e30d2304234d64b1afd0a7c46d"
server_key_version: "encryptionkey@0"
I20260924 07:06:12.754637 9485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "5e6e2025d7fb45d39df61ae084707a1c"
format_stamp: "Formatted at 2026-09-24 07:06:12 on dist-test-slave-2p7j"
server_key: "6d5f23655bafea01dead494f5fe533a0"
server_key_iv: "23fce1e30d2304234d64b1afd0a7c46d"
server_key_version: "encryptionkey@0"
I20260924 07:06:12.758193 9485 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 07:06:12.760576 9500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:12.761802 9485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:06:12.761909 9485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "5e6e2025d7fb45d39df61ae084707a1c"
format_stamp: "Formatted at 2026-09-24 07:06:12 on dist-test-slave-2p7j"
server_key: "6d5f23655bafea01dead494f5fe533a0"
server_key_iv: "23fce1e30d2304234d64b1afd0a7c46d"
server_key_version: "encryptionkey@0"
I20260924 07:06:12.761993 9485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:12.779201 9485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:12.782301 9485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:12.782455 9485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:12.789553 9485 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:37431
I20260924 07:06:12.790262 9552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:37431 every 8 connection(s)
I20260924 07:06:12.790458 9485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:06:12.794129 9553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:12.799392 9553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e6e2025d7fb45d39df61ae084707a1c: Bootstrap starting.
I20260924 07:06:12.800462 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 9485
I20260924 07:06:12.800582 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:06:12.800849 3940 external_mini_cluster.cc:1428] Setting key 4775094f7185c02bf487636575cf198a
I20260924 07:06:12.801982 9553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e6e2025d7fb45d39df61ae084707a1c: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:12.802806 9553 log.cc:824] T 00000000000000000000000000000000 P 5e6e2025d7fb45d39df61ae084707a1c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:12.804679 9553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e6e2025d7fb45d39df61ae084707a1c: No bootstrap required, opened a new log
I20260924 07:06:12.808038 9553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e6e2025d7fb45d39df61ae084707a1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e6e2025d7fb45d39df61ae084707a1c" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37431 } }
I20260924 07:06:12.808430 9553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e6e2025d7fb45d39df61ae084707a1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:12.808561 9553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e6e2025d7fb45d39df61ae084707a1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e6e2025d7fb45d39df61ae084707a1c, State: Initialized, Role: FOLLOWER
I20260924 07:06:12.809180 9553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e6e2025d7fb45d39df61ae084707a1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e6e2025d7fb45d39df61ae084707a1c" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37431 } }
I20260924 07:06:12.809381 9553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e6e2025d7fb45d39df61ae084707a1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:06:12.809456 9553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e6e2025d7fb45d39df61ae084707a1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:06:12.809572 9553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e6e2025d7fb45d39df61ae084707a1c [term 0 FOLLOWER]: Advancing to term 1
Sep 24 07:06:12 dist-test-slave-2p7j krb5kdc[9472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:06:12.810863 9553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e6e2025d7fb45d39df61ae084707a1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e6e2025d7fb45d39df61ae084707a1c" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37431 } }
I20260924 07:06:12.811329 9553 leader_election.cc:302] T 00000000000000000000000000000000 P 5e6e2025d7fb45d39df61ae084707a1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e6e2025d7fb45d39df61ae084707a1c; no voters:
I20260924 07:06:12.811828 9553 leader_election.cc:288] T 00000000000000000000000000000000 P 5e6e2025d7fb45d39df61ae084707a1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:06:12.811936 9558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e6e2025d7fb45d39df61ae084707a1c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:12.812141 9558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e6e2025d7fb45d39df61ae084707a1c [term 1 LEADER]: Becoming Leader. State: Replica: 5e6e2025d7fb45d39df61ae084707a1c, State: Running, Role: LEADER
I20260924 07:06:12.812520 9558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e6e2025d7fb45d39df61ae084707a1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e6e2025d7fb45d39df61ae084707a1c" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37431 } }
I20260924 07:06:12.813529 9553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e6e2025d7fb45d39df61ae084707a1c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:06:12.814493 9560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e6e2025d7fb45d39df61ae084707a1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e6e2025d7fb45d39df61ae084707a1c. Latest consensus state: current_term: 1 leader_uuid: "5e6e2025d7fb45d39df61ae084707a1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e6e2025d7fb45d39df61ae084707a1c" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37431 } } }
I20260924 07:06:12.814649 9560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e6e2025d7fb45d39df61ae084707a1c [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:12.815047 9567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:06:12.815582 9559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e6e2025d7fb45d39df61ae084707a1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e6e2025d7fb45d39df61ae084707a1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e6e2025d7fb45d39df61ae084707a1c" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37431 } } }
I20260924 07:06:12.815706 9559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e6e2025d7fb45d39df61ae084707a1c [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:12.816817 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:12.802276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34306 (local address 127.3.217.62:37431)
0924 07:06:12.802745 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:12.802800 (+ 55us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:12.802841 (+ 41us) server_negotiation.cc:407] Connection header received
0924 07:06:12.803632 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:12.803650 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:12.803997 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:12.804307 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:12.805121 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:12.806068 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:12.806870 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:12.807321 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:12.807459 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:12.810421 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:12.810426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:12.810438 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:12.810465 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:12.812966 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:12.813384 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:12.813390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:12.813396 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:12.813519 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:12.813792 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:12.813796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:12.813797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:12.814312 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:12.814512 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:12.815591 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0924 07:06:12.815896 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":122,"threads_started":1}
I20260924 07:06:12.819463 9567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:06:12.827225 9567 catalog_manager.cc:1347] Generated new cluster ID: 0147ecaba9194f77899d6eb1c57cdbdb
I20260924 07:06:12.827296 9567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:06:12.864319 9567 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:06:12.865643 9567 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:06:12.878170 9567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e6e2025d7fb45d39df61ae084707a1c: Generated new TSK 0
I20260924 07:06:12.878937 9567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
I20260924 07:06:12.911602 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:37431
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:33051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:13.031855 9579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:13.032099 9579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:13.032162 9579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:13.036087 9579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:13.036142 9579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:13.036221 9579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:06:13.041368 9579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:33051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:37431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:13.042339 9579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:13.043305 9579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:13.052297 9587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:13.052373 9585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:13.052299 9584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:13.052613 9579 server_base.cc:1034] running on GCE node
I20260924 07:06:13.052978 9579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:13.053527 9579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:13.054692 9579 hybrid_clock.cc:648] HybridClock initialized: now 1790233573054673 us; error 29 us; skew 500 ppm
Sep 24 07:06:13 dist-test-slave-2p7j krb5kdc[9472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233573, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:13.057693 9579 init.cc:377] Logged in from keytab as kudu/127.3.217.1@KRBTEST.COM (short username kudu)
I20260924 07:06:13.058950 9579 webserver.cc:460] Webserver started at http://127.3.217.1:38955/ using document root <none> and password file <none>
I20260924 07:06:13.059684 9579 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:13.059772 9579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:13.059973 9579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:13.062072 9579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "c39ff330d99f43d881c14d3fb67b25d0"
format_stamp: "Formatted at 2026-09-24 07:06:13 on dist-test-slave-2p7j"
server_key: "11fafd81f1c8dba482752d0d9ce496cf"
server_key_iv: "84940b8c5399a4e197f5ad412f66914d"
server_key_version: "encryptionkey@0"
I20260924 07:06:13.062567 9579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "c39ff330d99f43d881c14d3fb67b25d0"
format_stamp: "Formatted at 2026-09-24 07:06:13 on dist-test-slave-2p7j"
server_key: "11fafd81f1c8dba482752d0d9ce496cf"
server_key_iv: "84940b8c5399a4e197f5ad412f66914d"
server_key_version: "encryptionkey@0"
I20260924 07:06:13.066032 9579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 07:06:13.068270 9594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:13.069555 9579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:06:13.069670 9579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "c39ff330d99f43d881c14d3fb67b25d0"
format_stamp: "Formatted at 2026-09-24 07:06:13 on dist-test-slave-2p7j"
server_key: "11fafd81f1c8dba482752d0d9ce496cf"
server_key_iv: "84940b8c5399a4e197f5ad412f66914d"
server_key_version: "encryptionkey@0"
I20260924 07:06:13.069756 9579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:13.085031 9579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:13.087889 9579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:13.088063 9579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:13.088584 9579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:13.089385 9579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:13.089429 9579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:13.089480 9579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:13.089509 9579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:13.100729 9579 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:41609
I20260924 07:06:13.100765 9707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:41609 every 8 connection(s)
I20260924 07:06:13.101629 9579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:06:13.109625 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 9579
I20260924 07:06:13.109843 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:06:13.110177 3940 external_mini_cluster.cc:1428] Setting key 3bd0d7abdbe2f18ea85f0727b6cebce5
Sep 24 07:06:13 dist-test-slave-2p7j krb5kdc[9472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233573, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:06:13.117342 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:13.103553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:47765 (local address 127.3.217.62:37431)
0924 07:06:13.103687 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:13.103692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:13.104467 (+ 775us) server_negotiation.cc:407] Connection header received
0924 07:06:13.105759 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:13.105762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:13.105806 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:13.105879 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:13.107065 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:13.107808 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:13.109516 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:13.109991 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:13.110120 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:13.113334 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:13.113339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:13.113342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:13.113383 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:13.115168 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:13.115773 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:13.115777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:13.115778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:13.115863 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:13.116214 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:13.116218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:13.116220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:13.116414 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:13.116495 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:13.117037 (+ 542us) server_negotiation.cc:299] Negotiation successful
0924 07:06:13.117170 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:06:13.118093 9710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:13.103904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37431 (local address 127.3.217.1:47765)
0924 07:06:13.104326 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:13.104367 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:13.105559 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:13.106027 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:13.106033 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:13.106362 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:13.106823 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:13.106835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:13.107999 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:13.108003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:13.109306 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:13.109326 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:13.110186 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:13.110190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:13.110521 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:13.111525 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:13.111545 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:13.113133 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:13.115287 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:13.115292 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:13.115305 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:13.115639 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:13.115956 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:13.115959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:13.115961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:13.116114 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:13.116514 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:13.116520 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:13.116798 (+ 278us) client_negotiation.cc:769] Sending connection context
0924 07:06:13.116963 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 07:06:13.117279 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":102,"threads_started":1}
I20260924 07:06:13.119311 9708 heartbeater.cc:347] Connected to a master server at 127.3.217.62:37431
I20260924 07:06:13.119577 9708 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:13.120122 9708 heartbeater.cc:511] Master 127.3.217.62:37431 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.217.2@KRBTEST.COM; defaulting to no policy
I20260924 07:06:13.121620 9509 ts_manager.cc:194] Registered new tserver with Master: c39ff330d99f43d881c14d3fb67b25d0 (127.3.217.1:41609)
I20260924 07:06:13.122680 9509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.1@KRBTEST.COM'} at 127.3.217.1:47765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
I20260924 07:06:13.140031 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:37431
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:33051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:13.250578 9713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:13.251076 9713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:13.251210 9713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:13.256948 9713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:13.257069 9713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:13.257172 9713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:06:13.264461 9713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:33051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:37431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:13.265450 9713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:13.266425 9713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:13.275676 9721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:13.275779 9718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:13.275754 9713 server_base.cc:1034] running on GCE node
W20260924 07:06:13.275679 9719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:13.276401 9713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:13.277144 9713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:13.278370 9713 hybrid_clock.cc:648] HybridClock initialized: now 1790233573278350 us; error 43 us; skew 500 ppm
Sep 24 07:06:13 dist-test-slave-2p7j krb5kdc[9472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233573, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:13.281263 9713 init.cc:377] Logged in from keytab as kudu/127.3.217.2@KRBTEST.COM (short username kudu)
I20260924 07:06:13.282124 9713 webserver.cc:460] Webserver started at http://127.3.217.2:43415/ using document root <none> and password file <none>
I20260924 07:06:13.282657 9713 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:13.282721 9713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:13.282894 9713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:13.284859 9713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "556c46df9c684a55a7a0a253877cb75f"
format_stamp: "Formatted at 2026-09-24 07:06:13 on dist-test-slave-2p7j"
server_key: "09922786ef4aa8dc19086a7e32a910bd"
server_key_iv: "1119b97419449b579373569823acdaab"
server_key_version: "encryptionkey@0"
I20260924 07:06:13.285378 9713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "556c46df9c684a55a7a0a253877cb75f"
format_stamp: "Formatted at 2026-09-24 07:06:13 on dist-test-slave-2p7j"
server_key: "09922786ef4aa8dc19086a7e32a910bd"
server_key_iv: "1119b97419449b579373569823acdaab"
server_key_version: "encryptionkey@0"
I20260924 07:06:13.288902 9713 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:06:13.291760 9728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:13.292971 9713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:06:13.293077 9713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "556c46df9c684a55a7a0a253877cb75f"
format_stamp: "Formatted at 2026-09-24 07:06:13 on dist-test-slave-2p7j"
server_key: "09922786ef4aa8dc19086a7e32a910bd"
server_key_iv: "1119b97419449b579373569823acdaab"
server_key_version: "encryptionkey@0"
I20260924 07:06:13.293171 9713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:13.343782 9713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:13.347348 9713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:13.347537 9713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:13.348099 9713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:13.348896 9713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:13.348941 9713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:13.349001 9713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:13.349028 9713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:13.357407 9713 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:33117
I20260924 07:06:13.357566 9841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:33117 every 8 connection(s)
I20260924 07:06:13.358234 9713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:06:13.358551 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 9713
I20260924 07:06:13.358650 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:06:13.358925 3940 external_mini_cluster.cc:1428] Setting key 23b80dacc56082f63322405418833a97
WARNING: no policy specified for kudu/127.3.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.3@KRBTEST.COM" created.
Sep 24 07:06:13 dist-test-slave-2p7j krb5kdc[9472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233573, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:06:13.374565 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:13.360040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:44919 (local address 127.3.217.62:37431)
0924 07:06:13.360166 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:13.360170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:13.361084 (+ 914us) server_negotiation.cc:407] Connection header received
0924 07:06:13.362670 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:13.362674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:13.362723 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:13.362808 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:13.363996 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:13.364789 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:13.366354 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:13.366727 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:13.366836 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:13.371289 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:13.371292 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:13.371294 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:13.371315 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:13.372563 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:13.373038 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:13.373090 (+ 52us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:13.373092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:13.373178 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:13.373515 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:13.373518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:13.373520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:13.373687 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:13.373770 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:13.374273 (+ 503us) server_negotiation.cc:299] Negotiation successful
0924 07:06:13.374402 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:06:13.375312 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:13.360459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37431 (local address 127.3.217.2:44919)
0924 07:06:13.360924 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:13.360977 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:13.362480 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:13.363025 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:13.363031 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:13.363359 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:13.363863 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:13.363876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:13.365025 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:13.365031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:13.366228 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:13.366241 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:13.366879 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:13.366882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:13.367020 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:13.367624 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:13.367640 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:13.371137 (+ 3497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:13.372670 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:13.372675 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:13.372689 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:13.372950 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:13.373287 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:13.373290 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:13.373292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:13.373426 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:13.373781 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:13.373787 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:13.374038 (+ 251us) client_negotiation.cc:769] Sending connection context
0924 07:06:13.374220 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 07:06:13.374524 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":95,"threads_started":1}
I20260924 07:06:13.376374 9842 heartbeater.cc:347] Connected to a master server at 127.3.217.62:37431
I20260924 07:06:13.376618 9842 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:13.377102 9842 heartbeater.cc:511] Master 127.3.217.62:37431 requested a full tablet report, sending...
I20260924 07:06:13.378160 9509 ts_manager.cc:194] Registered new tserver with Master: 556c46df9c684a55a7a0a253877cb75f (127.3.217.2:33117)
I20260924 07:06:13.378831 9509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.2@KRBTEST.COM'} at 127.3.217.2:44919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
I20260924 07:06:13.386184 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:37431
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:33051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:13.494836 9847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:13.495077 9847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:13.495116 9847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:13.499079 9847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:13.499143 9847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:13.499199 9847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:06:13.504310 9847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:33051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:37431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:13.505239 9847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:13.506150 9847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:06:13.516075 9847 server_base.cc:1034] running on GCE node
W20260924 07:06:13.516047 9853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:13.516111 9852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:13.516225 9855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:13.516600 9847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:13.517201 9847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:13.518378 9847 hybrid_clock.cc:648] HybridClock initialized: now 1790233573518360 us; error 31 us; skew 500 ppm
Sep 24 07:06:13 dist-test-slave-2p7j krb5kdc[9472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233573, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:13.521044 9847 init.cc:377] Logged in from keytab as kudu/127.3.217.3@KRBTEST.COM (short username kudu)
I20260924 07:06:13.521869 9847 webserver.cc:460] Webserver started at http://127.3.217.3:37761/ using document root <none> and password file <none>
I20260924 07:06:13.522369 9847 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:13.522426 9847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:13.522598 9847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:13.524544 9847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "90fcf4403d274a26a0b5a2ec87ef32bd"
format_stamp: "Formatted at 2026-09-24 07:06:13 on dist-test-slave-2p7j"
server_key: "c3cc681a6877f865adbcd9b69f6da82d"
server_key_iv: "4c1a367127f78a8e70584b7c95881776"
server_key_version: "encryptionkey@0"
I20260924 07:06:13.525041 9847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "90fcf4403d274a26a0b5a2ec87ef32bd"
format_stamp: "Formatted at 2026-09-24 07:06:13 on dist-test-slave-2p7j"
server_key: "c3cc681a6877f865adbcd9b69f6da82d"
server_key_iv: "4c1a367127f78a8e70584b7c95881776"
server_key_version: "encryptionkey@0"
I20260924 07:06:13.528906 9847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:06:13.532104 9862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:13.533329 9847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:06:13.533469 9847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "90fcf4403d274a26a0b5a2ec87ef32bd"
format_stamp: "Formatted at 2026-09-24 07:06:13 on dist-test-slave-2p7j"
server_key: "c3cc681a6877f865adbcd9b69f6da82d"
server_key_iv: "4c1a367127f78a8e70584b7c95881776"
server_key_version: "encryptionkey@0"
I20260924 07:06:13.533577 9847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:13.558411 9847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:13.562448 9847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:13.562640 9847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:13.563302 9847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:13.564260 9847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:13.564322 9847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:13.564394 9847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:13.564424 9847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:13.575111 9847 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:40025
I20260924 07:06:13.575160 9975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:40025 every 8 connection(s)
I20260924 07:06:13.576102 9847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:06:13.582507 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 9847
I20260924 07:06:13.582612 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:06:13.582883 3940 external_mini_cluster.cc:1428] Setting key e9e64230425dd24f8796f39cb5478207
Sep 24 07:06:13 dist-test-slave-2p7j krb5kdc[9472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233573, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:06:13.592651 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:13.579474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:51291 (local address 127.3.217.62:37431)
0924 07:06:13.579629 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:13.579634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:13.579652 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:06:13.580865 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:13.580869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:13.580936 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:13.581032 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:13.582347 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:13.583158 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:13.585070 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:13.585417 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:13.585608 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:13.588656 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:13.588660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:13.588663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:13.588696 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:13.590037 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:13.590619 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:13.590622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:13.590623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:13.590696 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:13.591179 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:13.591185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:13.591186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:13.591574 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:13.591720 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:13.592369 (+ 649us) server_negotiation.cc:299] Negotiation successful
0924 07:06:13.592499 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 07:06:13.593595 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:13.578512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37431 (local address 127.3.217.3:51291)
0924 07:06:13.579032 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:13.579087 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:13.580635 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:13.581195 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:13.581204 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:13.581591 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:13.582218 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:13.582239 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:13.583305 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:13.583309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:13.584929 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:13.584941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:13.585646 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:13.585650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:13.585859 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:13.586553 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:13.586584 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:13.588495 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:13.590145 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:13.590151 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:13.590167 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:13.590510 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:13.590772 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:13.590775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:13.590777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:13.590943 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:13.591719 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:13.591726 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:13.592028 (+ 302us) client_negotiation.cc:769] Sending connection context
0924 07:06:13.592247 (+ 219us) client_negotiation.cc:240] Negotiation successful
0924 07:06:13.592594 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":130,"threads_started":1}
I20260924 07:06:13.594934 9976 heartbeater.cc:347] Connected to a master server at 127.3.217.62:37431
I20260924 07:06:13.595176 9976 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:13.595677 9976 heartbeater.cc:511] Master 127.3.217.62:37431 requested a full tablet report, sending...
I20260924 07:06:13.596751 9509 ts_manager.cc:194] Registered new tserver with Master: 90fcf4403d274a26a0b5a2ec87ef32bd (127.3.217.3:40025)
I20260924 07:06:13.597518 9509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.3@KRBTEST.COM'} at 127.3.217.3:51291
I20260924 07:06:13.600816 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/test-admin.keytab.
Sep 24 07:06:13 dist-test-slave-2p7j krb5kdc[9472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:13.620983 3940 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 07:06:13.623987 9983 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:13.622295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43258 (local address 127.3.217.1:41609)
0924 07:06:13.622712 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:13.622721 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:13.622749 (+ 28us) server_negotiation.cc:407] Connection header received
0924 07:06:13.622811 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:13.622816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:13.623002 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:13.623143 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:13.623864 (+ 721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43258: BlockingRecv error: recv got EOF from 127.0.0.1:43258 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":95,"threads_started":1}
W20260924 07:06:13.624375 3940 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/krb5cc)
Sep 24 07:06:13 dist-test-slave-2p7j krb5kdc[9472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:13.625736 3940 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 07:06:13 dist-test-slave-2p7j krb5kdc[9472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:13.627014 3940 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 07:06:13 dist-test-slave-2p7j krb5kdc[9472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.1@KRBTEST.COM
I20260924 07:06:13.637966 9983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:13.628343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.3.217.1:41609)
0924 07:06:13.628497 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:13.628502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:13.628514 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:06:13.628546 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:13.628548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:13.628584 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:13.628648 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:13.629172 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:13.629944 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:13.631209 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:13.631672 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:13.631781 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:13.634406 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:13.634422 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:13.634445 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:13.634483 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:13.636261 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:13.636677 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:13.636682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:13.636684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:13.636766 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:13.636999 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:13.637003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:13.637004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:13.637284 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:13.637500 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:13.637674 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 07:06:13.637829 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 07:06:13.642621 9983 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:13.641583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43268 (local address 127.3.217.1:41609)
0924 07:06:13.641729 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:13.641733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:13.641895 (+ 162us) server_negotiation.cc:407] Connection header received
0924 07:06:13.641996 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:13.641999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:13.642036 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:13.642097 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:13.642541 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43268: BlockingRecv error: recv got EOF from 127.0.0.1:43268 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 07:06:13.642874 3940 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/krb5cc' of type FILE
Sep 24 07:06:13 dist-test-slave-2p7j krb5kdc[9472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:13.643886 3940 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 07:06:13 dist-test-slave-2p7j krb5kdc[9472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:13.644744 3940 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 07:06:13 dist-test-slave-2p7j krb5kdc[9472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.1@KRBTEST.COM
I20260924 07:06:13.652648 9983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:13.645550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43270 (local address 127.3.217.1:41609)
0924 07:06:13.645654 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:13.645658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:13.645727 (+ 69us) server_negotiation.cc:407] Connection header received
0924 07:06:13.645863 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:13.645865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:13.645898 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:13.645954 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:13.646416 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:13.646908 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:13.647693 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:13.647919 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:13.647999 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:13.649970 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:13.649973 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:13.649975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:13.649995 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:13.651196 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:13.651614 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:13.651617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:13.651619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:13.651688 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:13.651928 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:13.651931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:13.651933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:13.652098 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:13.652224 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:13.652419 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 07:06:13.652516 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 07:06:13.655467 9983 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:13.654619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.3.217.1:41609)
0924 07:06:13.654744 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:13.654748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:13.654841 (+ 93us) server_negotiation.cc:407] Connection header received
0924 07:06:13.654942 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:13.654945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:13.654983 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:13.655042 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:13.655392 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43278: BlockingRecv error: recv got EOF from 127.0.0.1:43278 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 07:06:13.655673 3940 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233541052846-3940-0/krb5kdc/krb5cc' of type FILE
Sep 24 07:06:13 dist-test-slave-2p7j krb5kdc[9472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:13.656571 3940 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 07:06:13 dist-test-slave-2p7j krb5kdc[9472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:13.657943 3940 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 07:06:13 dist-test-slave-2p7j krb5kdc[9472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.1@KRBTEST.COM
I20260924 07:06:13.666399 9983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:13.658714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43280 (local address 127.3.217.1:41609)
0924 07:06:13.658858 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:13.658862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:13.658925 (+ 63us) server_negotiation.cc:407] Connection header received
0924 07:06:13.659025 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:13.659028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:13.659062 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:13.659125 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:13.659586 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:13.660060 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:13.660787 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:13.660983 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:13.661061 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:13.663281 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:13.663284 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:13.663285 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:13.663304 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:13.664520 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:13.665188 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:13.665191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:13.665193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:13.665265 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:13.665636 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:13.665641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:13.665642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:13.665817 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:13.665943 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:13.666152 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 07:06:13.666248 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 07:06:13.667331 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 9579
I20260924 07:06:13.673143 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 9713
I20260924 07:06:13.677645 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 9847
I20260924 07:06:13.682528 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 9485
2026-09-24T07:06:13Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2157 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:06:13 dist-test-slave-2p7j krb5kdc[9996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:06:13 dist-test-slave-2p7j krb5kdc[9996](info): set up 2 sockets
Sep 24 07:06:13 dist-test-slave-2p7j krb5kdc[9996](info): commencing operation
krb5kdc: starting...
W20260924 07:06:14.660370 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:06:14 dist-test-slave-2p7j krb5kdc[9996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:06:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:06:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.62.keytab.
Entry for principal oryx/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.62.keytab.
I20260924 07:06:14.800000 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:36411
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.62.keytab
--principal=oryx/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:44339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:36411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:14.914211 10009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:14.914894 10009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:14.914974 10009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:14.920542 10009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:06:14.920622 10009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:14.920641 10009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:14.920660 10009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:06:14.920675 10009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:06:14.926321 10009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:44339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:36411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.62.keytab
--principal=oryx/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:36411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:14.927240 10009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:14.928170 10009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:14.936141 10014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:14.936189 10015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:14.936146 10017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:14.936275 10009 server_base.cc:1034] running on GCE node
I20260924 07:06:14.936903 10009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:14.937845 10009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:14.939042 10009 hybrid_clock.cc:648] HybridClock initialized: now 1790233574938960 us; error 94 us; skew 500 ppm
Sep 24 07:06:14 dist-test-slave-2p7j krb5kdc[9996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233574, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:14.941640 10009 init.cc:377] Logged in from keytab as oryx/127.3.217.62@KRBTEST.COM (short username oryx)
I20260924 07:06:14.942423 10009 webserver.cc:460] Webserver started at http://127.3.217.62:35095/ using document root <none> and password file <none>
I20260924 07:06:14.942911 10009 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:14.942972 10009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:14.943141 10009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:14.945212 10009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "8c46e717f5ae4e6a9c5e6849154e5295"
format_stamp: "Formatted at 2026-09-24 07:06:14 on dist-test-slave-2p7j"
server_key: "bc3ea360c6d7bc9e2eb93846b92b3cad"
server_key_iv: "ff4dbde3d72f4eaa818e3cd34215faaf"
server_key_version: "encryptionkey@0"
I20260924 07:06:14.945747 10009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "8c46e717f5ae4e6a9c5e6849154e5295"
format_stamp: "Formatted at 2026-09-24 07:06:14 on dist-test-slave-2p7j"
server_key: "bc3ea360c6d7bc9e2eb93846b92b3cad"
server_key_iv: "ff4dbde3d72f4eaa818e3cd34215faaf"
server_key_version: "encryptionkey@0"
I20260924 07:06:14.949533 10009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 07:06:14.951813 10024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:14.953632 10009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:06:14.953868 10009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "8c46e717f5ae4e6a9c5e6849154e5295"
format_stamp: "Formatted at 2026-09-24 07:06:14 on dist-test-slave-2p7j"
server_key: "bc3ea360c6d7bc9e2eb93846b92b3cad"
server_key_iv: "ff4dbde3d72f4eaa818e3cd34215faaf"
server_key_version: "encryptionkey@0"
I20260924 07:06:14.954017 10009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:14.967381 10009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:14.970413 10009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:14.970597 10009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:14.977825 10009 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:36411
I20260924 07:06:14.978240 10076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:36411 every 8 connection(s)
I20260924 07:06:14.978735 10009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:06:14.981616 10077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:14.986371 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 10009
I20260924 07:06:14.986502 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:06:14.986768 3940 external_mini_cluster.cc:1428] Setting key 9614894aecfd96b40493126c93011687
I20260924 07:06:14.986997 10077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295: Bootstrap starting.
I20260924 07:06:14.989566 10077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:14.990410 10077 log.cc:824] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:14.992414 10077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295: No bootstrap required, opened a new log
Sep 24 07:06:14 dist-test-slave-2p7j krb5kdc[9996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM
I20260924 07:06:14.996160 10077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } }
I20260924 07:06:14.996359 10077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:14.996400 10077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c46e717f5ae4e6a9c5e6849154e5295, State: Initialized, Role: FOLLOWER
I20260924 07:06:14.997079 10077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } }
I20260924 07:06:14.997336 10077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:06:14.997419 10077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:06:14.997540 10077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:14.998884 10077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } }
I20260924 07:06:14.999404 10077 leader_election.cc:302] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c46e717f5ae4e6a9c5e6849154e5295; no voters:
I20260924 07:06:14.999812 10077 leader_election.cc:288] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:06:14.999913 10082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:15.000123 10082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [term 1 LEADER]: Becoming Leader. State: Replica: 8c46e717f5ae4e6a9c5e6849154e5295, State: Running, Role: LEADER
I20260924 07:06:15.000460 10082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } }
I20260924 07:06:15.001097 10077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:06:15.001543 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:14.988156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50128 (local address 127.3.217.62:36411)
0924 07:06:14.988568 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:14.988585 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:14.988624 (+ 39us) server_negotiation.cc:407] Connection header received
0924 07:06:14.989379 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:14.989400 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:14.989817 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:14.990147 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:14.990780 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:14.992109 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:14.992993 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:14.993647 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:14.993809 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:14.996594 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:14.996600 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:14.996611 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:14.996639 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:14.998314 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:14.998738 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:14.998754 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:14.998758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:14.998957 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:14.999249 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:14.999253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:14.999254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:14.999694 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:14.999870 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:15.000512 (+ 642us) server_negotiation.cc:299] Negotiation successful
0924 07:06:15.000775 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":109,"threads_started":1}
I20260924 07:06:15.001657 10082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c46e717f5ae4e6a9c5e6849154e5295. Latest consensus state: current_term: 1 leader_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } }
I20260924 07:06:15.001786 10082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:15.002030 10084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } }
I20260924 07:06:15.002138 10084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [sys.catalog]: This master's current role is: LEADER
W20260924 07:06:15.005952 10097 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:06:15.006036 10097 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:06:15.006124 10087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:06:15.006887 10087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:06:15.013960 10087 catalog_manager.cc:1347] Generated new cluster ID: 5730979573034234b8f25895f606f0bd
I20260924 07:06:15.014036 10087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:06:15.024050 10087 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:06:15.025262 10087 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:06:15.042833 10087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295: Generated new TSK 0
I20260924 07:06:15.044281 10087 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.1.keytab.
Entry for principal oryx/127.3.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.1.keytab.
I20260924 07:06:15.087848 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:36411
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.1.keytab
--principal=oryx/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:44339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:15.197969 10103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:15.198210 10103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:15.198251 10103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:15.202317 10103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:15.202373 10103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:15.202432 10103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:06:15.207564 10103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:44339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.1.keytab
--principal=oryx/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:36411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:15.208529 10103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:15.209461 10103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:15.218628 10108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:15.218667 10109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:15.218631 10111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:15.218788 10103 server_base.cc:1034] running on GCE node
I20260924 07:06:15.219183 10103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:15.219785 10103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:15.220964 10103 hybrid_clock.cc:648] HybridClock initialized: now 1790233575220940 us; error 35 us; skew 500 ppm
Sep 24 07:06:15 dist-test-slave-2p7j krb5kdc[9996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233575, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:15.223747 10103 init.cc:377] Logged in from keytab as oryx/127.3.217.1@KRBTEST.COM (short username oryx)
I20260924 07:06:15.224521 10103 webserver.cc:460] Webserver started at http://127.3.217.1:43471/ using document root <none> and password file <none>
I20260924 07:06:15.224990 10103 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:15.225042 10103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:15.225186 10103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:15.226997 10103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "921e678ee49a47b5a1be6be3174a856d"
format_stamp: "Formatted at 2026-09-24 07:06:15 on dist-test-slave-2p7j"
server_key: "611d537244c4350595fb96b06b56d481"
server_key_iv: "c4bf72eaa7a1465b431b5f98be721647"
server_key_version: "encryptionkey@0"
I20260924 07:06:15.227473 10103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "921e678ee49a47b5a1be6be3174a856d"
format_stamp: "Formatted at 2026-09-24 07:06:15 on dist-test-slave-2p7j"
server_key: "611d537244c4350595fb96b06b56d481"
server_key_iv: "c4bf72eaa7a1465b431b5f98be721647"
server_key_version: "encryptionkey@0"
I20260924 07:06:15.230795 10103 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:06:15.233386 10118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:15.234740 10103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 07:06:15.234853 10103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "921e678ee49a47b5a1be6be3174a856d"
format_stamp: "Formatted at 2026-09-24 07:06:15 on dist-test-slave-2p7j"
server_key: "611d537244c4350595fb96b06b56d481"
server_key_iv: "c4bf72eaa7a1465b431b5f98be721647"
server_key_version: "encryptionkey@0"
I20260924 07:06:15.234965 10103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:15.255405 10103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:15.259416 10103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:15.259626 10103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:15.260248 10103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:15.261077 10103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:15.261125 10103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:15.261178 10103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:15.261207 10103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:15.271148 10103 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:37477
I20260924 07:06:15.271462 10231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:37477 every 8 connection(s)
I20260924 07:06:15.272038 10103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:06:15.273756 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 10103
I20260924 07:06:15.273873 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:06:15.274125 3940 external_mini_cluster.cc:1428] Setting key 4b3779586eee1f2fbfd1bc9a417cfeab
Sep 24 07:06:15 dist-test-slave-2p7j krb5kdc[9996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233575, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.3.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.217.2@KRBTEST.COM" created.
I20260924 07:06:15.288539 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:15.274553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:54859 (local address 127.3.217.62:36411)
0924 07:06:15.274688 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:15.274692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:15.275899 (+ 1207us) server_negotiation.cc:407] Connection header received
0924 07:06:15.276876 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:15.276879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:15.276928 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:15.277031 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:15.278767 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:15.279291 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:15.280895 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:15.281120 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:15.281203 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:15.284124 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:15.284127 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:15.284129 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:15.284152 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:15.285916 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:15.286488 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:15.286493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:15.286494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:15.286641 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:15.287132 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:15.287135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:15.287137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:15.287337 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:15.287451 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:15.288209 (+ 758us) server_negotiation.cc:299] Negotiation successful
0924 07:06:15.288380 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:06:15.289220 10234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:15.274654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.3.217.1:54859)
0924 07:06:15.275401 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:15.275470 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:15.276654 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:15.277328 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:15.277336 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:15.277821 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:15.278586 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:15.278605 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:15.279421 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:15.279425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:15.280743 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:15.280754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:15.281280 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:15.281285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:15.281481 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:15.282145 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:15.282167 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:15.283934 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:15.286025 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:15.286030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:15.286042 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:15.286292 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:15.286815 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:15.286818 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:15.286820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:15.286996 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:15.287547 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:15.287554 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:15.287853 (+ 299us) client_negotiation.cc:769] Sending connection context
0924 07:06:15.288059 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 07:06:15.288334 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":182,"threads_started":1}
I20260924 07:06:15.290843 10232 heartbeater.cc:347] Connected to a master server at 127.3.217.62:36411
I20260924 07:06:15.291316 10232 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:15.292254 10232 heartbeater.cc:511] Master 127.3.217.62:36411 requested a full tablet report, sending...
I20260924 07:06:15.294421 10041 ts_manager.cc:194] Registered new tserver with Master: 921e678ee49a47b5a1be6be3174a856d (127.3.217.1:37477)
I20260924 07:06:15.295581 10041 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.217.1@KRBTEST.COM'} at 127.3.217.1:54859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.2.keytab.
Entry for principal oryx/127.3.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.2.keytab.
I20260924 07:06:15.312304 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:36411
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.2.keytab
--principal=oryx/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:44339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:15.426863 10237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:15.427106 10237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:15.427163 10237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:15.431085 10237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:15.431135 10237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:15.431209 10237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:06:15.436511 10237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:44339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.2.keytab
--principal=oryx/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:36411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:15.437413 10237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:15.438370 10237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:15.447384 10245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:15.447395 10242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:15.447396 10243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:15.447624 10237 server_base.cc:1034] running on GCE node
I20260924 07:06:15.447992 10237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:15.448570 10237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:15.449747 10237 hybrid_clock.cc:648] HybridClock initialized: now 1790233575449648 us; error 112 us; skew 500 ppm
Sep 24 07:06:15 dist-test-slave-2p7j krb5kdc[9996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233575, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:15.452608 10237 init.cc:377] Logged in from keytab as oryx/127.3.217.2@KRBTEST.COM (short username oryx)
I20260924 07:06:15.453846 10237 webserver.cc:460] Webserver started at http://127.3.217.2:40975/ using document root <none> and password file <none>
I20260924 07:06:15.454340 10237 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:15.454417 10237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:15.454645 10237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:15.456563 10237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "6eafe362852c419ba36a9014bffa7f03"
format_stamp: "Formatted at 2026-09-24 07:06:15 on dist-test-slave-2p7j"
server_key: "492aebc166d308e24a33d5904efe7b45"
server_key_iv: "8d76d2f63ae32548082f6f9bd261ff9f"
server_key_version: "encryptionkey@0"
I20260924 07:06:15.457088 10237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "6eafe362852c419ba36a9014bffa7f03"
format_stamp: "Formatted at 2026-09-24 07:06:15 on dist-test-slave-2p7j"
server_key: "492aebc166d308e24a33d5904efe7b45"
server_key_iv: "8d76d2f63ae32548082f6f9bd261ff9f"
server_key_version: "encryptionkey@0"
I20260924 07:06:15.460651 10237 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:06:15.462893 10252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:15.464178 10237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:06:15.464273 10237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "6eafe362852c419ba36a9014bffa7f03"
format_stamp: "Formatted at 2026-09-24 07:06:15 on dist-test-slave-2p7j"
server_key: "492aebc166d308e24a33d5904efe7b45"
server_key_iv: "8d76d2f63ae32548082f6f9bd261ff9f"
server_key_version: "encryptionkey@0"
I20260924 07:06:15.464373 10237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:15.490140 10237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:15.493057 10237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:15.493223 10237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:15.493775 10237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:15.494558 10237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:15.494606 10237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:15.494656 10237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:15.494684 10237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:15.503053 10237 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:45971
I20260924 07:06:15.503099 10365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:45971 every 8 connection(s)
I20260924 07:06:15.503935 10237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:06:15.509725 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 10237
I20260924 07:06:15.509835 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:06:15.510113 3940 external_mini_cluster.cc:1428] Setting key 6300c1eb4cf922c86019ffba64d4516f
Sep 24 07:06:15 dist-test-slave-2p7j krb5kdc[9996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233575, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM
I20260924 07:06:15.518308 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:15.505578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:60793 (local address 127.3.217.62:36411)
0924 07:06:15.505687 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:15.505691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:15.506441 (+ 750us) server_negotiation.cc:407] Connection header received
0924 07:06:15.507650 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:15.507653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:15.507696 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:15.507756 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:15.508790 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:15.509303 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:15.511190 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:15.511537 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:15.511646 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:15.514687 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:15.514690 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:15.514692 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:15.514713 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:15.516001 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:15.516542 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:15.516545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:15.516546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:15.516616 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:15.516975 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:15.516977 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:15.516978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:15.517134 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:15.517206 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:15.517961 (+ 755us) server_negotiation.cc:299] Negotiation successful
0924 07:06:15.518143 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:06:15.519155 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:15.505866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.3.217.2:60793)
0924 07:06:15.506301 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:15.506339 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:15.507410 (+ 1071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:15.507880 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:15.507886 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:15.508206 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:15.508651 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:15.508663 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:15.509487 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:15.509491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:15.510976 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:15.511001 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:15.511699 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:15.511704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:15.512113 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:15.512727 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:15.512748 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:15.514491 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:15.516111 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:15.516118 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:15.516133 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:15.516451 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:15.516700 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:15.516703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:15.516705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:15.516888 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:15.517235 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:15.517241 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:15.517548 (+ 307us) client_negotiation.cc:769] Sending connection context
0924 07:06:15.517777 (+ 229us) client_negotiation.cc:240] Negotiation successful
0924 07:06:15.518144 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for oryx/127.3.217.3@KRBTEST.COM; defaulting to no policy
I20260924 07:06:15.520403 10366 heartbeater.cc:347] Connected to a master server at 127.3.217.62:36411
I20260924 07:06:15.520623 10366 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:15.521185 10366 heartbeater.cc:511] Master 127.3.217.62:36411 requested a full tablet report, sending...
I20260924 07:06:15.522279 10041 ts_manager.cc:194] Registered new tserver with Master: 6eafe362852c419ba36a9014bffa7f03 (127.3.217.2:45971)
I20260924 07:06:15.522848 10041 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.217.2@KRBTEST.COM'} at 127.3.217.2:60793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.3.keytab.
Entry for principal oryx/127.3.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.3.keytab.
I20260924 07:06:15.539352 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:36411
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.3.keytab
--principal=oryx/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:44339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:15.651225 10371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:15.651485 10371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:15.651564 10371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:15.655488 10371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:15.655565 10371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:15.655624 10371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:06:15.663128 10371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:44339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.3.keytab
--principal=oryx/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:36411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:15.664569 10371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:15.665555 10371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:15.675489 10379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:15.675459 10377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:15.675635 10371 server_base.cc:1034] running on GCE node
W20260924 07:06:15.675431 10376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:15.676743 10371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:15.677343 10371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:15.678547 10371 hybrid_clock.cc:648] HybridClock initialized: now 1790233575678510 us; error 66 us; skew 500 ppm
Sep 24 07:06:15 dist-test-slave-2p7j krb5kdc[9996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233575, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:15.681128 10371 init.cc:377] Logged in from keytab as oryx/127.3.217.3@KRBTEST.COM (short username oryx)
I20260924 07:06:15.681921 10371 webserver.cc:460] Webserver started at http://127.3.217.3:41561/ using document root <none> and password file <none>
I20260924 07:06:15.682433 10371 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:15.682492 10371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:15.682642 10371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:15.684558 10371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4"
format_stamp: "Formatted at 2026-09-24 07:06:15 on dist-test-slave-2p7j"
server_key: "18368c47c9c46c0bcaa3a93e3fd529d4"
server_key_iv: "3a93109d106a73400669ae2b95b08ef3"
server_key_version: "encryptionkey@0"
I20260924 07:06:15.685067 10371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4"
format_stamp: "Formatted at 2026-09-24 07:06:15 on dist-test-slave-2p7j"
server_key: "18368c47c9c46c0bcaa3a93e3fd529d4"
server_key_iv: "3a93109d106a73400669ae2b95b08ef3"
server_key_version: "encryptionkey@0"
I20260924 07:06:15.688580 10371 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 07:06:15.691248 10386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:15.692767 10371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:06:15.692889 10371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4"
format_stamp: "Formatted at 2026-09-24 07:06:15 on dist-test-slave-2p7j"
server_key: "18368c47c9c46c0bcaa3a93e3fd529d4"
server_key_iv: "3a93109d106a73400669ae2b95b08ef3"
server_key_version: "encryptionkey@0"
I20260924 07:06:15.692984 10371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:15.713450 10371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:15.717197 10371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:15.717386 10371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:15.718104 10371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:15.718914 10371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:15.718961 10371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:15.719017 10371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:15.719046 10371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:15.728319 10371 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:36493
I20260924 07:06:15.728351 10499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:36493 every 8 connection(s)
I20260924 07:06:15.729146 10371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:06:15.736079 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 10371
I20260924 07:06:15.736268 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:06:15.736523 3940 external_mini_cluster.cc:1428] Setting key 321ca66de3ee4621e089831415ff03fe
Sep 24 07:06:15 dist-test-slave-2p7j krb5kdc[9996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233575, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM
I20260924 07:06:15.745265 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:15.730962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:39723 (local address 127.3.217.62:36411)
0924 07:06:15.731133 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:15.731138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:15.732577 (+ 1439us) server_negotiation.cc:407] Connection header received
0924 07:06:15.733587 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:15.733591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:15.733644 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:15.733728 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:15.734914 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:15.735774 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:15.737669 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:15.737886 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:15.737976 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:15.740846 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:15.740850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:15.740852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:15.740893 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:15.742380 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:15.743362 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:15.743366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:15.743368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:15.743490 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:15.744141 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:15.744144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:15.744147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:15.744377 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:15.744476 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:15.744983 (+ 507us) server_negotiation.cc:299] Negotiation successful
0924 07:06:15.745101 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 07:06:15.746031 10502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:15.731645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.3.217.3:39723)
0924 07:06:15.732157 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:15.732198 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:15.733350 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:15.733868 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:15.733874 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:15.734199 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:15.734677 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:15.734689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:15.735918 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:15.735922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:15.737427 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:15.737439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:15.738135 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:15.738140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:15.738285 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:15.738935 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:15.738951 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:15.740635 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:15.742553 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:15.742559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:15.742573 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:15.743180 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:15.743666 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:15.743669 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:15.743671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:15.743931 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:15.744494 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:15.744501 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:15.744745 (+ 244us) client_negotiation.cc:769] Sending connection context
0924 07:06:15.744917 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 07:06:15.745236 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":126,"threads_started":1}
I20260924 07:06:15.747418 10500 heartbeater.cc:347] Connected to a master server at 127.3.217.62:36411
I20260924 07:06:15.747677 10500 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:15.748139 10500 heartbeater.cc:511] Master 127.3.217.62:36411 requested a full tablet report, sending...
I20260924 07:06:15.749501 10041 ts_manager.cc:194] Registered new tserver with Master: 25fe6ef8b01840ecb7cfae4d871a0ba4 (127.3.217.3:36493)
I20260924 07:06:15.750109 10041 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.217.3@KRBTEST.COM'} at 127.3.217.3:39723
I20260924 07:06:15.750973 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.217.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.217.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.217.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.61.keytab.
Entry for principal oryx/127.3.217.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.61.keytab.
I20260924 07:06:15.787228 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.61:44701
--webserver_interface=127.3.217.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.61.keytab
--principal=oryx/127.3.217.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:44339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:36411,127.3.217.61:44701
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:15.903038 10505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:15.903276 10505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:15.903314 10505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:15.907660 10505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:06:15.907727 10505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:15.907745 10505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:15.907765 10505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:06:15.907780 10505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:06:15.913245 10505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:44339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:36411,127.3.217.61:44701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.61.keytab
--principal=oryx/127.3.217.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.61:44701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.217.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
W20260924 07:06:15.913599 10505 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.217.62:36411,127.3.217.61:44701'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 07:06:15.914951 10505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:15.916044 10505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:15.925316 10511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:15.925343 10513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:15.925323 10510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:15.925678 10505 server_base.cc:1034] running on GCE node
I20260924 07:06:15.926195 10505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:15.927240 10505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:15.928399 10505 hybrid_clock.cc:648] HybridClock initialized: now 1790233575928379 us; error 33 us; skew 500 ppm
Sep 24 07:06:15 dist-test-slave-2p7j krb5kdc[9996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233575, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:15.931030 10505 init.cc:377] Logged in from keytab as oryx/127.3.217.61@KRBTEST.COM (short username oryx)
I20260924 07:06:15.931942 10505 webserver.cc:460] Webserver started at http://127.3.217.61:41841/ using document root <none> and password file <none>
I20260924 07:06:15.932454 10505 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:15.932516 10505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:15.932689 10505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:15.934598 10505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/data/instance:
uuid: "2c9e3ac96bc54ab2a397e3e94d422074"
format_stamp: "Formatted at 2026-09-24 07:06:15 on dist-test-slave-2p7j"
server_key: "967fa9061e00e5ed0b7990490e2c42e8"
server_key_iv: "74b293782d7b53eb72217a61ab8cf4e6"
server_key_version: "encryptionkey@0"
I20260924 07:06:15.935096 10505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/wal/instance:
uuid: "2c9e3ac96bc54ab2a397e3e94d422074"
format_stamp: "Formatted at 2026-09-24 07:06:15 on dist-test-slave-2p7j"
server_key: "967fa9061e00e5ed0b7990490e2c42e8"
server_key_iv: "74b293782d7b53eb72217a61ab8cf4e6"
server_key_version: "encryptionkey@0"
I20260924 07:06:15.938963 10505 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:06:15.942248 10520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:15.944043 10505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:06:15.944170 10505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/wal
uuid: "2c9e3ac96bc54ab2a397e3e94d422074"
format_stamp: "Formatted at 2026-09-24 07:06:15 on dist-test-slave-2p7j"
server_key: "967fa9061e00e5ed0b7990490e2c42e8"
server_key_iv: "74b293782d7b53eb72217a61ab8cf4e6"
server_key_version: "encryptionkey@0"
I20260924 07:06:15.944298 10505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:15.964179 10505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:15.968240 10505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:15.968434 10505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:15.975279 10505 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.61:44701
I20260924 07:06:15.975749 10572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.61:44701 every 8 connection(s)
I20260924 07:06:15.976253 10505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/data/info.pb
I20260924 07:06:15.979545 10573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:15.984143 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 10505
I20260924 07:06:15.984476 10573 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:06:15 dist-test-slave-2p7j krb5kdc[9996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233575, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.61@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM
Sep 24 07:06:15 dist-test-slave-2p7j krb5kdc[9996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.217.61@KRBTEST.COM
I20260924 07:06:15.998731 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:15.986647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50140 (local address 127.3.217.62:36411)
0924 07:06:15.986757 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:15.986761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:15.987235 (+ 474us) server_negotiation.cc:407] Connection header received
0924 07:06:15.987649 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:15.987652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:15.987702 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:15.987769 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:15.988772 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:15.989285 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:15.990994 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:15.991295 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:15.991402 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:15.993840 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:15.993843 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:15.993845 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:15.993866 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:15.995493 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:15.996487 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:15.996493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:15.996495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:15.996690 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:15.997150 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:15.997154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:15.997156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:15.997553 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:15.997669 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:15.998363 (+ 694us) server_negotiation.cc:299] Negotiation successful
0924 07:06:15.998570 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:06:15.999562 10577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:15.986585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.0.0.1:50140)
0924 07:06:15.987103 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:15.987150 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:15.987446 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:15.987930 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:15.987935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:15.988330 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:15.988657 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:15.988668 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:15.989396 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:15.989414 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:15.990795 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:15.990817 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:15.991468 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:15.991472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:15.991713 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:15.992252 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:15.992272 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:15.993693 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:15.995633 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:15.995642 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:15.995652 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:15.996300 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:15.996858 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:15.996862 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:15.996864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:15.997011 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:15.997719 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:15.997725 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:15.998089 (+ 364us) client_negotiation.cc:769] Sending connection context
0924 07:06:15.998262 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 07:06:15.998588 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":183,"thread_start_us":76,"threads_started":1}
I20260924 07:06:15.999693 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:15.986030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39730 (local address 127.3.217.61:44701)
0924 07:06:15.986423 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:15.986439 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:15.986477 (+ 38us) server_negotiation.cc:407] Connection header received
0924 07:06:15.987298 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:15.987319 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:15.987591 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:15.987753 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:15.988599 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:15.990037 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:15.991360 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:15.991824 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:15.991926 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:15.995137 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:15.995153 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:15.995166 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:15.995193 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:15.997545 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:15.998115 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:15.998121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:15.998122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:15.998249 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:15.998606 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:15.998611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:15.998613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:15.998901 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:15.999124 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:15.999347 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 07:06:15.999488 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":259,"thread_start_us":103,"threads_started":1}
I20260924 07:06:16.001017 10573 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:06:16 dist-test-slave-2p7j krb5kdc[9996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233575, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.61@KRBTEST.COM for oryx/127.3.217.61@KRBTEST.COM
I20260924 07:06:16.010108 10577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.001492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:44701 (local address 127.0.0.1:39740)
0924 07:06:16.001586 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:16.001599 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:16.001669 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:16.002021 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:16.002024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:16.002151 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:16.002226 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.002230 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.002952 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.002955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.003464 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.003472 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.004330 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.004333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.004460 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.004839 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:16.004849 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:16.006408 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:16.008367 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.008370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.008371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.008542 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.008994 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.008997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.008999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.009089 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.009668 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:16.009671 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:16.009746 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 07:06:16.009834 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 07:06:16.009946 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":18}
I20260924 07:06:16.010138 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.001567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39740 (local address 127.3.217.61:44701)
0924 07:06:16.001872 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:16.001875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:16.001884 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:06:16.001908 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:16.001909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:16.001938 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:16.001995 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:16.002319 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.002840 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.003603 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.004235 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.004324 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.006559 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:16.006562 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:16.006564 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:16.006583 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:16.008247 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.008732 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.008737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.008739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.008869 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.009209 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.009212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.009214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.009554 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:16.009646 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:16.009876 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 07:06:16.010001 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":252}
I20260924 07:06:16.013316 10573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074: Bootstrap starting.
I20260924 07:06:16.015622 10573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:16.016300 10573 log.cc:824] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:16.019273 10573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074: No bootstrap required, opened a new log
I20260924 07:06:16.023113 10573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } }
I20260924 07:06:16.023320 10573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:16.023353 10573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c9e3ac96bc54ab2a397e3e94d422074, State: Initialized, Role: FOLLOWER
I20260924 07:06:16.023828 10573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } }
I20260924 07:06:16.024312 10579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } } }
I20260924 07:06:16.024425 10579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:06:16.025069 10573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 07:06:16.029860 10590 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:06:16.029942 10590 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:06:16.032819 10505 master_runner.cc:428] Detected that this master 2c9e3ac96bc54ab2a397e3e94d422074 is joining an existing cluster
I20260924 07:06:16.032882 10505 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.217.61:44701
I20260924 07:06:16.046640 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.036532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50142 (local address 127.3.217.62:36411)
0924 07:06:16.036670 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:16.036674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:16.036967 (+ 293us) server_negotiation.cc:407] Connection header received
0924 07:06:16.037627 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:16.037631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:16.037685 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:16.037738 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:16.038937 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.039822 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.040714 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.040940 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.041033 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.042561 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:16.042566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:16.042568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:16.042593 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:16.044631 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.045285 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.045288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.045290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.045391 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.045671 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.045674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.045676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.045893 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:16.045984 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:16.046322 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 07:06:16.046436 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 07:06:16.046830 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.036089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39756 (local address 127.3.217.61:44701)
0924 07:06:16.036204 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:16.036208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:16.037153 (+ 945us) server_negotiation.cc:407] Connection header received
0924 07:06:16.037800 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:16.037809 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:16.037869 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:16.037953 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:16.038860 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.039710 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.041476 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.041852 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.041942 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.043818 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:16.043822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:16.043824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:16.043849 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:16.045421 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.045832 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.045836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.045837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.045900 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.046111 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.046114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.046115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.046269 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:16.046345 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:16.046553 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 07:06:16.046691 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":48,"spinlock_wait_cycles":1920}
I20260924 07:06:16.046840 10597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.036714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:44701 (local address 127.0.0.1:39756)
0924 07:06:16.037071 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:16.037083 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:16.037405 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:16.037985 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:16.037988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:16.038592 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:16.038747 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.038754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.039851 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.039855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.041352 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.041363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.041976 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.041980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.042083 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.042920 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:16.042933 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:16.043702 (+ 769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:16.045511 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.045514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.045515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.045720 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.045974 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.045977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.045979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.046043 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.046345 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:16.046348 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:16.046427 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 07:06:16.046506 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 07:06:16.046639 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":60,"spinlock_wait_cycles":453504,"thread_start_us":101,"threads_started":1}
I20260924 07:06:16.047081 10596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.036063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.0.0.1:50142)
0924 07:06:16.036827 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:16.036847 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:16.037355 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:16.037928 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:16.037932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:16.038591 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:16.038773 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.038780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.039919 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.039922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.040528 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.040536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.041042 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.041045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.041145 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.042050 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:16.042060 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:16.042429 (+ 369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:16.044732 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.044735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.044737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.045090 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.045487 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.045490 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.045492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.045591 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.046029 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:16.046033 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:16.046147 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 07:06:16.046775 (+ 628us) client_negotiation.cc:240] Negotiation successful
0924 07:06:16.046910 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":23,"spinlock_wait_cycles":1792,"thread_start_us":123,"threads_started":1}
Sep 24 07:06:16 dist-test-slave-2p7j krb5kdc[9996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233574, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.62@KRBTEST.COM for oryx/127.3.217.61@KRBTEST.COM
WARNING: no policy specified for oryx/127.3.217.60@KRBTEST.COM; defaulting to no policy
I20260924 07:06:16.066763 10598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.054735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:44701 (local address 127.0.0.1:39770)
0924 07:06:16.055773 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:16.055821 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:16.055985 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:16.056546 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:16.056553 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:16.057020 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:16.057146 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.057153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.058427 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.058430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.059151 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.059160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.059629 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.059632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.059730 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.060203 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:16.060215 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:16.062078 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:16.064998 (+ 2920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.065009 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.065016 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.065254 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.065614 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.065617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.065619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.065708 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.066101 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:16.066109 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:16.066228 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 07:06:16.066364 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 07:06:16.066583 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"thread_start_us":101,"threads_started":1}
I20260924 07:06:16.066761 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.054804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39770 (local address 127.3.217.61:44701)
0924 07:06:16.054936 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:16.054941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:16.055915 (+ 974us) server_negotiation.cc:407] Connection header received
0924 07:06:16.056132 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:16.056135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:16.056285 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:16.056528 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:16.057271 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.058215 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.059262 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.059496 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.059623 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.062218 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:16.062221 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:16.062224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:16.062245 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:16.064861 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.065416 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.065420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.065421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.065502 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.065815 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.065818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.065819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.065994 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:16.066078 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:16.066428 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 07:06:16.066617 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:06:16.068527 10041 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.217.61:44701: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074"
member_type: NON_VOTER
last_known_addr {
host: "127.3.217.61"
port: 44701
}
attrs {
promote: true
}
}
dest_uuid: "8c46e717f5ae4e6a9c5e6849154e5295"
cas_config_opid_index: -1
I20260924 07:06:16.069208 10041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: NON_VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.217.60@KRBTEST.COM" created.
I20260924 07:06:16.071869 10547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:16.073364 10601 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2c9e3ac96bc54ab2a397e3e94d422074 (127.3.217.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: NON_VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: true } } }
I20260924 07:06:16.073791 10547 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [term 1 FOLLOWER]: Refusing update from remote peer 8c46e717f5ae4e6a9c5e6849154e5295: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 07:06:16.074594 10603 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: NON_VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:06:16.075220 10601 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.217.61:44701
I20260924 07:06:16.075654 10505 master.cc:537] Master@127.3.217.61:44701 shutting down...
W20260924 07:06:16.076329 10026 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 07:06:16.077221 10604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: NON_VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: true } } }
I20260924 07:06:16.077322 10604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [sys.catalog]: This master's current role is: LEADER
W20260924 07:06:16.077313 10026 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 -> Peer 2c9e3ac96bc54ab2a397e3e94d422074 (127.3.217.61:44701): Couldn't send request to peer 2c9e3ac96bc54ab2a397e3e94d422074. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 07:06:16.079767 10505 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:06:16.080170 10505 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:16.080283 10505 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.217.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.60.keytab.
Entry for principal oryx/127.3.217.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.60.keytab.
I20260924 07:06:16.095242 10505 master.cc:552] Master@127.3.217.61:44701 shutdown complete.
I20260924 07:06:16.095305 10505 master_runner.cc:305] Clearing existing system tablet
I20260924 07:06:16.095577 10505 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:16.095633 10505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:16.096131 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.60:38295
--webserver_interface=127.3.217.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.60.keytab
--principal=oryx/127.3.217.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:44339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:36411,127.3.217.61:44701,127.3.217.60:38295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:06:16.096817 10505 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:06:16.097600 10609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:16.097800 10505 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:16.097903 10505 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:06:16.097934 10505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/wal
uuid: "2c9e3ac96bc54ab2a397e3e94d422074"
format_stamp: "Formatted at 2026-09-24 07:06:15 on dist-test-slave-2p7j"
server_key: "967fa9061e00e5ed0b7990490e2c42e8"
server_key_iv: "74b293782d7b53eb72217a61ab8cf4e6"
server_key_version: "encryptionkey@0"
I20260924 07:06:16.100245 10505 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:06:16.102808 10505 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 07:06:16.103304 10505 log.cc:1196] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 07:06:16.103636 10505 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074: Deleting consensus metadata
I20260924 07:06:16.103820 10505 master_runner.cc:315] Copying system tablet from 127.3.217.62:36411
I20260924 07:06:16.104976 10505 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074: tablet copy: Beginning tablet copy session from remote peer at address 127.3.217.62:36411
I20260924 07:06:16.116858 10615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.105507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.0.0.1:50156)
0924 07:06:16.107699 (+ 2192us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:16.107713 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:16.107829 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:16.108089 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:16.108092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:16.108306 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:16.108431 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.108438 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.109479 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.109483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.110608 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.110636 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.111602 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.111607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.111786 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.112645 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:16.112659 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:16.113064 (+ 405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:16.115318 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.115321 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.115324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.115601 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.115869 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.115872 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.115874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.115956 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.116337 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:16.116340 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:16.116433 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 07:06:16.116523 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 07:06:16.116656 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2115,"thread_start_us":82,"threads_started":1}
I20260924 07:06:16.116926 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.105605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50156 (local address 127.3.217.62:36411)
0924 07:06:16.105732 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:16.105737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:16.107798 (+ 2061us) server_negotiation.cc:407] Connection header received
0924 07:06:16.107912 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:16.107915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:16.107966 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:16.108064 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:16.108543 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.109374 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.110971 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.111374 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.111571 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.113273 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:16.113278 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:16.113281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:16.113317 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:16.115197 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.115690 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.115693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.115696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.115784 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.116039 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.116042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.116044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.116256 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:16.116341 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:16.116603 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 07:06:16.116717 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:06:16.117998 10061 tablet_copy_service.cc:140] P 8c46e717f5ae4e6a9c5e6849154e5295: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2c9e3ac96bc54ab2a397e3e94d422074 ({username='oryx', principal='oryx/127.3.217.61@KRBTEST.COM'} at 127.0.0.1:50156)
I20260924 07:06:16.118180 10061 tablet_copy_service.cc:161] P 8c46e717f5ae4e6a9c5e6849154e5295: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2c9e3ac96bc54ab2a397e3e94d422074 at {username='oryx', principal='oryx/127.3.217.61@KRBTEST.COM'} at 127.0.0.1:50156: session id = 2c9e3ac96bc54ab2a397e3e94d422074-00000000000000000000000000000000
I20260924 07:06:16.119932 10061 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 07:06:16.121531 10505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 07:06:16.123837 10505 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074: tablet copy: Starting download of 0 data blocks...
I20260924 07:06:16.124063 10505 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074: tablet copy: Starting download of 1 WAL segments...
I20260924 07:06:16.126384 10505 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 07:06:16.128827 10061 tablet_copy_service.cc:342] P 8c46e717f5ae4e6a9c5e6849154e5295: Request end of tablet copy session 2c9e3ac96bc54ab2a397e3e94d422074-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.217.61@KRBTEST.COM'} at 127.0.0.1:50156
I20260924 07:06:16.128962 10061 tablet_copy_service.cc:433] P 8c46e717f5ae4e6a9c5e6849154e5295: ending tablet copy session 2c9e3ac96bc54ab2a397e3e94d422074-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2c9e3ac96bc54ab2a397e3e94d422074
I20260924 07:06:16.136807 10505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:16.138835 10616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:16.138885 10619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:16.139185 10505 server_base.cc:1034] running on GCE node
W20260924 07:06:16.151909 10617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:16.152197 10505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:16.152524 10505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:16.153663 10505 hybrid_clock.cc:648] HybridClock initialized: now 1790233576153644 us; error 39 us; skew 500 ppm
Sep 24 07:06:16 dist-test-slave-2p7j krb5kdc[9996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233576, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:16.155454 10505 init.cc:377] Logged in from keytab as oryx/127.3.217.61@KRBTEST.COM (short username oryx)
I20260924 07:06:16.155946 10505 webserver.cc:460] Webserver started at http://127.3.217.61:33297/ using document root <none> and password file <none>
I20260924 07:06:16.156136 10505 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:16.156183 10505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:16.157631 10505 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:06:16.158660 10626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:16.158917 10505 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:06:16.158998 10505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/wal
uuid: "2c9e3ac96bc54ab2a397e3e94d422074"
format_stamp: "Formatted at 2026-09-24 07:06:15 on dist-test-slave-2p7j"
server_key: "967fa9061e00e5ed0b7990490e2c42e8"
server_key_iv: "74b293782d7b53eb72217a61ab8cf4e6"
server_key_version: "encryptionkey@0"
I20260924 07:06:16.159081 10505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:16.187629 10505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:16.189615 10505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:16.194085 10505 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.61:44701
I20260924 07:06:16.194416 10678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.61:44701 every 8 connection(s)
I20260924 07:06:16.194787 10505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-1/data/info.pb
I20260924 07:06:16.196880 10679 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:06:16.197872 10679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074: Bootstrap starting.
I20260924 07:06:16.209185 10679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:06:16.209470 10679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074: Bootstrap complete.
I20260924 07:06:16.209985 10679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: NON_VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: true } }
I20260924 07:06:16.210111 10679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2c9e3ac96bc54ab2a397e3e94d422074, State: Initialized, Role: LEARNER
I20260924 07:06:16.210207 10679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: NON_VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: true } }
I20260924 07:06:16.210646 10679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:06:16.210615 10682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: NON_VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: true } } }
I20260924 07:06:16.210875 10682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [sys.catalog]: This master's current role is: LEARNER
I20260924 07:06:16.216336 10693 catalog_manager.cc:1259] Loaded cluster ID: 5730979573034234b8f25895f606f0bd
I20260924 07:06:16.216393 10693 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074: loading cluster ID for follower catalog manager: success
I20260924 07:06:16.218981 10693 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074: acquiring CA information for follower catalog manager: success
I20260924 07:06:16.220857 10693 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 07:06:16.234079 10608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:16.234331 10608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:16.234391 10608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:16.239266 10608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:06:16.239326 10608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:16.239353 10608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:16.239375 10608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:06:16.239413 10608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:06:16.245049 10608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:44339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:36411,127.3.217.61:44701,127.3.217.60:38295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.60.keytab
--principal=oryx/127.3.217.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.60:38295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.217.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:16.246075 10608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:16.247098 10608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:16.255692 10701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:16.255746 10698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:16.255710 10699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:16.255782 10608 server_base.cc:1034] running on GCE node
I20260924 07:06:16.256649 10608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:16.257817 10608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:16.259003 10608 hybrid_clock.cc:648] HybridClock initialized: now 1790233576258993 us; error 41 us; skew 500 ppm
Sep 24 07:06:16 dist-test-slave-2p7j krb5kdc[9996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233576, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:16.262006 10608 init.cc:377] Logged in from keytab as oryx/127.3.217.60@KRBTEST.COM (short username oryx)
I20260924 07:06:16.263213 10608 webserver.cc:460] Webserver started at http://127.3.217.60:38633/ using document root <none> and password file <none>
I20260924 07:06:16.263818 10608 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:16.263887 10608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:16.264072 10608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:16.266517 10608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/data/instance:
uuid: "8571268ea3b24b0ca48031986cc19c2e"
format_stamp: "Formatted at 2026-09-24 07:06:16 on dist-test-slave-2p7j"
server_key: "c58fb3a5fe443691daaafc1e3acbe731"
server_key_iv: "4a4c4ef03e920e24b13e98e0b3f9c692"
server_key_version: "encryptionkey@0"
I20260924 07:06:16.267666 10608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/wal/instance:
uuid: "8571268ea3b24b0ca48031986cc19c2e"
format_stamp: "Formatted at 2026-09-24 07:06:16 on dist-test-slave-2p7j"
server_key: "c58fb3a5fe443691daaafc1e3acbe731"
server_key_iv: "4a4c4ef03e920e24b13e98e0b3f9c692"
server_key_version: "encryptionkey@0"
I20260924 07:06:16.272846 10608 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260924 07:06:16.275964 10708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:16.277401 10608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 07:06:16.277522 10608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/wal
uuid: "8571268ea3b24b0ca48031986cc19c2e"
format_stamp: "Formatted at 2026-09-24 07:06:16 on dist-test-slave-2p7j"
server_key: "c58fb3a5fe443691daaafc1e3acbe731"
server_key_iv: "4a4c4ef03e920e24b13e98e0b3f9c692"
server_key_version: "encryptionkey@0"
I20260924 07:06:16.277643 10608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:16.294137 10608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:16.298045 10608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:16.298246 10608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:16.299414 10232 heartbeater.cc:503] Master 127.3.217.62:36411 was elected leader, sending a full tablet report...
I20260924 07:06:16.305500 10608 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.60:38295
I20260924 07:06:16.305802 10760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.60:38295 every 8 connection(s)
I20260924 07:06:16.306504 10608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/data/info.pb
I20260924 07:06:16.309693 10761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:16.313365 10761 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:06:16.313650 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 10608
Sep 24 07:06:16 dist-test-slave-2p7j krb5kdc[9996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233576, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.60@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM
Sep 24 07:06:16 dist-test-slave-2p7j krb5kdc[9996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.217.60@KRBTEST.COM
I20260924 07:06:16.325807 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.315293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50172 (local address 127.3.217.62:36411)
0924 07:06:16.315410 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:16.315415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:16.315791 (+ 376us) server_negotiation.cc:407] Connection header received
0924 07:06:16.316607 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:16.316609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:16.316653 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:16.316727 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:16.317587 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.318207 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.319228 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.319445 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.319571 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.322087 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:16.322091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:16.322094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:16.322119 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:16.323886 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.324339 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.324343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.324345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.324428 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.324773 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.324776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.324777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.324949 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:16.325020 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:16.325541 (+ 521us) server_negotiation.cc:299] Negotiation successful
0924 07:06:16.325649 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:06:16.326905 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.315127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.0.0.1:50172)
0924 07:06:16.315648 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:16.315694 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:16.316395 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:16.316777 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:16.316782 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:16.317191 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:16.317428 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.317442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.318332 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.318335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.319104 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.319114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.319593 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.319597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.319771 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.320417 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:16.320432 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:16.321924 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:16.323979 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.323985 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.323993 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.324241 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.324535 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.324538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.324539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.324681 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.325041 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:16.325048 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:16.325300 (+ 252us) client_negotiation.cc:769] Sending connection context
0924 07:06:16.325480 (+ 180us) client_negotiation.cc:240] Negotiation successful
0924 07:06:16.325703 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":49,"thread_start_us":142,"threads_started":1}
I20260924 07:06:16.326922 10763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.315127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.3.217.60:38295)
0924 07:06:16.315656 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:16.315673 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:16.315710 (+ 37us) server_negotiation.cc:407] Connection header received
0924 07:06:16.316385 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:16.316407 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:16.316633 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:16.316779 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:16.317441 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.318356 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.319156 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.319550 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.319763 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.322529 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:16.322535 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:16.322547 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:16.322571 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:16.324606 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.325097 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.325101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.325102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.325166 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.325415 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.325417 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.325418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.325660 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:16.326076 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:16.326406 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 07:06:16.326520 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":430,"spinlock_wait_cycles":45440,"thread_start_us":142,"threads_started":1}
I20260924 07:06:16.328284 10761 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:06:16 dist-test-slave-2p7j krb5kdc[9996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233576, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.60@KRBTEST.COM for oryx/127.3.217.61@KRBTEST.COM
I20260924 07:06:16.336678 10765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.328835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39784 (local address 127.3.217.61:44701)
0924 07:06:16.329067 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:16.329071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:16.329081 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:06:16.329112 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:16.329114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:16.329150 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:16.329220 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:16.329568 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.330217 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.330921 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.331130 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.331209 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.333284 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:16.333287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:16.333289 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:16.333308 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:16.334631 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.335013 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.335016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.335017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.335092 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.335753 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.335757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.335759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.336019 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:16.336126 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:16.336441 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 07:06:16.336552 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":87,"threads_started":1}
I20260924 07:06:16.336870 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.328721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:44701 (local address 127.0.0.1:39784)
0924 07:06:16.328853 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:16.328863 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:16.328915 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:16.329246 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:16.329249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:16.329389 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:16.329471 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.329475 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.330311 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.330315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.330826 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.330833 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.331220 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.331223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.331311 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.331779 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:16.331795 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:16.333157 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:16.334728 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.334731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.334732 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.334905 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.335181 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.335184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.335185 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.335389 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.336150 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:16.336154 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:16.336229 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 07:06:16.336331 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 07:06:16.336575 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 07:06:16.337589 10761 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.60" port: 38295 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:06:16 dist-test-slave-2p7j krb5kdc[9996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233576, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.60@KRBTEST.COM for oryx/127.3.217.60@KRBTEST.COM
I20260924 07:06:16.346462 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.338215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:38295 (local address 127.0.0.1:52832)
0924 07:06:16.338354 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:16.338368 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:16.338466 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:16.338819 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:16.338821 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:16.338956 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:16.339027 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.339032 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.339847 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.339850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.340354 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.340362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.341410 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.341413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.341512 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.341968 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:16.341979 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:16.343305 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:16.345097 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.345100 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.345101 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.345273 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.345534 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.345537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.345538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.345610 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.346103 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:16.346106 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:16.346178 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 07:06:16.346261 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:06:16.346357 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 07:06:16.346612 10763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.338295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52832 (local address 127.3.217.60:38295)
0924 07:06:16.338400 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:16.338404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:16.338434 (+ 30us) server_negotiation.cc:407] Connection header received
0924 07:06:16.338607 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:16.338610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:16.338647 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:16.338708 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:16.339156 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.339744 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.340590 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.341254 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.341345 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.343533 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:16.343537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:16.343540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:16.343563 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:16.345003 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.345370 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.345373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.345374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.345448 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.345718 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.345721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.345722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.345880 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:16.346104 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:16.346386 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 07:06:16.346469 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:06:16.349193 10761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e: Bootstrap starting.
I20260924 07:06:16.351274 10761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:16.352047 10761 log.cc:824] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:16.354204 10761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e: No bootstrap required, opened a new log
I20260924 07:06:16.357110 10761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 38295 } }
I20260924 07:06:16.357290 10761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:16.357331 10761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8571268ea3b24b0ca48031986cc19c2e, State: Initialized, Role: FOLLOWER
I20260924 07:06:16.357709 10761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 38295 } }
I20260924 07:06:16.358191 10767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 38295 } } }
I20260924 07:06:16.358316 10767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:06:16.358839 10761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [sys.catalog]: configured and running, proceeding with master startup.
W20260924 07:06:16.364392 10778 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:06:16.364475 10778 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:06:16.367944 10608 master_runner.cc:428] Detected that this master 8571268ea3b24b0ca48031986cc19c2e is joining an existing cluster
I20260924 07:06:16.368018 10608 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.217.60:38295
I20260924 07:06:16.378753 10765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.370589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47644 (local address 127.3.217.61:44701)
0924 07:06:16.370717 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:16.370721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:16.370847 (+ 126us) server_negotiation.cc:407] Connection header received
0924 07:06:16.370942 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:16.370945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:16.370989 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:16.371059 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:16.371621 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.372395 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.374022 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.374235 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.374329 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.375458 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:16.375462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:16.375465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:16.375488 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:16.377031 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.377427 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.377431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.377433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.377510 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.377767 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.377771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.377773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.377976 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:16.378071 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:16.378483 (+ 412us) server_negotiation.cc:299] Negotiation successful
0924 07:06:16.378600 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:06:16.379287 10785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.369964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:44701 (local address 127.0.0.1:47644)
0924 07:06:16.370754 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:16.370767 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:16.370865 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:16.371094 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:16.371097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:16.371335 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:16.371459 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.371467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.372604 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.372607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.373906 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.373916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.374356 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.374360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.374451 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.374913 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:16.374925 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:16.375301 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:16.377144 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.377148 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.377150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.377349 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.377585 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.377588 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.377590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.377673 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.378076 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:16.378079 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:16.378165 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 07:06:16.378461 (+ 296us) client_negotiation.cc:240] Negotiation successful
0924 07:06:16.378753 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":81,"threads_started":1}
I20260924 07:06:16.380216 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.369964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59906 (local address 127.3.217.62:36411)
0924 07:06:16.370079 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:16.370084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:16.370734 (+ 650us) server_negotiation.cc:407] Connection header received
0924 07:06:16.370940 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:16.370943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:16.370989 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:16.371037 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:16.371592 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.372401 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.375045 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.375359 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.375446 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.376583 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:16.376587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:16.376590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:16.376610 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:16.378096 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.378879 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.378882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.378884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.378971 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.379308 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.379312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.379314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.379584 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:16.379651 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:16.379969 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 07:06:16.380081 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:06:16.380405 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 10103
I20260924 07:06:16.380717 10786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.369910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.0.0.1:59906)
0924 07:06:16.370644 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:16.370657 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:16.370800 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:16.371186 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:16.371189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:16.371380 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:16.371472 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.371477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.374036 (+ 2559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.374039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.374932 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.374942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.375586 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.375589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.375660 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.376101 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:16.376114 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:16.376459 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:16.378199 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.378202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.378204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.378745 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.379085 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.379088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.379090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.379207 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.379717 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:16.379720 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:16.379800 (+ 80us) client_negotiation.cc:769] Sending connection context
0924 07:06:16.380377 (+ 577us) client_negotiation.cc:240] Negotiation successful
0924 07:06:16.380582 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":101,"threads_started":1}
I20260924 07:06:16.388127 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:37477
--local_ip_for_outbound_sockets=127.3.217.1
--tserver_master_addrs=127.3.217.62:36411
--webserver_port=43471
--webserver_interface=127.3.217.1
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.1.keytab
--principal=oryx/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:44339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.217.62:36411,127.3.217.61:44701
--tserver_master_addrs=127.3.217.62:36411,127.3.217.61:44701,127.3.217.60:38295 with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:06:16 dist-test-slave-2p7j krb5kdc[9996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233574, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.62@KRBTEST.COM for oryx/127.3.217.60@KRBTEST.COM
I20260924 07:06:16.411599 10598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.385440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:38295 (local address 127.0.0.1:52848)
0924 07:06:16.385536 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:16.385549 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:16.385616 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:16.385941 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:16.385945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:16.386098 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:16.386186 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.386191 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.387042 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.387046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.387672 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.387682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.388102 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.388106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.388206 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.388652 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:16.388663 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:16.389951 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:16.409416 (+ 19465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.409420 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.409423 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.409681 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.410003 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.410006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.410008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.410092 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.410513 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:16.410516 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:16.410809 (+ 293us) client_negotiation.cc:769] Sending connection context
0924 07:06:16.411025 (+ 216us) client_negotiation.cc:240] Negotiation successful
0924 07:06:16.411292 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 07:06:16.412245 10784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.370350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:38295 (local address 127.0.0.1:52842)
0924 07:06:16.380034 (+ 9684us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:16.380046 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:16.380151 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:16.380411 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:16.380413 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:16.381671 (+ 1258us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:16.381773 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.381781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.382780 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.382783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.383654 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.383664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.384115 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.384118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.384208 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.408205 (+ 23997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:16.408224 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:16.408997 (+ 773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:16.410732 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.410735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.410737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.410968 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.411229 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.411232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.411233 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.411301 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.411736 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:16.411738 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:16.411829 (+ 91us) client_negotiation.cc:769] Sending connection context
0924 07:06:16.411917 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 07:06:16.412057 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9618,"thread_start_us":90,"threads_started":1}
I20260924 07:06:16.412703 10763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.370435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52842 (local address 127.3.217.60:38295)
0924 07:06:16.370532 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:16.370535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:16.380107 (+ 9572us) server_negotiation.cc:407] Connection header received
0924 07:06:16.380231 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:16.380234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:16.380267 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:16.380308 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:16.381933 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.382589 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.383762 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.384022 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.384095 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.409165 (+ 25070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:16.409169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:16.409172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:16.409192 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:16.410611 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.411081 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.411083 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.411085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.411152 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.411392 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.411395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.411397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.411594 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:16.411676 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:16.412486 (+ 810us) server_negotiation.cc:299] Negotiation successful
0924 07:06:16.412586 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:06:16.413597 10787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.385505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52848 (local address 127.3.217.60:38295)
0924 07:06:16.385726 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:16.385732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:16.385742 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:06:16.385791 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:16.385793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:16.385829 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:16.385908 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:16.386297 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.386856 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.387782 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.388003 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.388085 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.390093 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:16.390097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:16.390099 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:16.390123 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:16.409272 (+ 19149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.409804 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.409807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.409809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.409902 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.410195 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.410198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.410200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.410412 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:16.413170 (+ 2758us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:16.413257 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 07:06:16.413403 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":98,"threads_started":1}
I20260924 07:06:16.414743 10041 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.217.60:38295: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e"
member_type: NON_VOTER
last_known_addr {
host: "127.3.217.60"
port: 38295
}
attrs {
promote: true
}
}
dest_uuid: "8c46e717f5ae4e6a9c5e6849154e5295"
cas_config_opid_index: 5
I20260924 07:06:16.415349 10041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: NON_VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: true } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: NON_VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: true } }
I20260924 07:06:16.416420 10603 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8571268ea3b24b0ca48031986cc19c2e (127.3.217.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: NON_VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: true } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: NON_VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: true } } }
I20260924 07:06:16.417299 10735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:16.417893 10603 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.217.60:38295
I20260924 07:06:16.418052 10603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: NON_VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: true } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: NON_VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: true } } }
I20260924 07:06:16.418139 10603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:16.418609 10735 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [term 1 FOLLOWER]: Refusing update from remote peer 8c46e717f5ae4e6a9c5e6849154e5295: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 07:06:16.419118 10608 master.cc:537] Master@127.3.217.60:38295 shutting down...
I20260924 07:06:16.419154 10604 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: NON_VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 07:06:16.421651 10025 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 -> Peer 8571268ea3b24b0ca48031986cc19c2e (127.3.217.60:38295): Couldn't send request to peer 8571268ea3b24b0ca48031986cc19c2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 07:06:16.422211 10598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.416579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:44701 (local address 127.0.0.1:47658)
0924 07:06:16.417048 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:16.417062 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:16.417138 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:16.417540 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:16.417543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:16.417561 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:16.417646 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.417652 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.418615 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.418618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.420086 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.420097 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.421812 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.421816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.421919 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.421959 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 07:06:16.422026 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 07:06:16.422093 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394}
I20260924 07:06:16.422731 10765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.416811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47658 (local address 127.3.217.61:44701)
0924 07:06:16.416959 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:16.416964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:16.417213 (+ 249us) server_negotiation.cc:407] Connection header received
0924 07:06:16.417265 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:16.417268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:16.417327 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:16.417417 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:16.417756 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.418490 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.420218 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.421537 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.421631 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.421753 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:16.422544 (+ 791us) server_negotiation.cc:299] Negotiation successful
0924 07:06:16.422616 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 07:06:16.423157 10653 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [term 1 LEARNER]: Refusing update from remote peer 8c46e717f5ae4e6a9c5e6849154e5295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 07:06:16.423678 10604 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: NON_VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 07:06:16.424021 10608 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:06:16.424273 10608 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:16.424368 10608 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e: stopping tablet replica
I20260924 07:06:16.425412 10653 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8571268ea3b24b0ca48031986cc19c2e (127.3.217.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: NON_VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: true } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: NON_VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: true } } }
I20260924 07:06:16.427193 10682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c46e717f5ae4e6a9c5e6849154e5295. Latest consensus state: current_term: 1 leader_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: NON_VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: true } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: NON_VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: true } } }
I20260924 07:06:16.427307 10682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [sys.catalog]: This master's current role is: LEARNER
I20260924 07:06:16.427389 10682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: NON_VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: true } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: NON_VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: true } } }
I20260924 07:06:16.427469 10682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [sys.catalog]: This master's current role is: LEARNER
I20260924 07:06:16.428644 10604 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295: attempting to promote NON_VOTER 2c9e3ac96bc54ab2a397e3e94d422074 to VOTER
I20260924 07:06:16.429075 10604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: NON_VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: true } }
I20260924 07:06:16.430160 10653 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [term 1 LEARNER]: Refusing update from remote peer 8c46e717f5ae4e6a9c5e6849154e5295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 07:06:16.430280 10598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.429993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:38295 (local address 127.0.0.1:52860)
0924 07:06:16.430144 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:16.430209 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 07:06:16.430581 10603 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 07:06:16.430971 10598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.430721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:38295 (local address 127.0.0.1:52876)
0924 07:06:16.430870 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:16.430914 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260924 07:06:16.431078 10025 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 -> Peer 8571268ea3b24b0ca48031986cc19c2e (127.3.217.60:38295): Couldn't send request to peer 8571268ea3b24b0ca48031986cc19c2e. Status: Network error: Client connection negotiation failed: client connection to 127.3.217.60:38295: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 07:06:16.432103 10601 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2c9e3ac96bc54ab2a397e3e94d422074 (127.3.217.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: NON_VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: true } } }
I20260924 07:06:16.433425 10601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: NON_VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: true } } }
I20260924 07:06:16.433521 10601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:16.433624 10601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: NON_VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: true } } }
I20260924 07:06:16.433692 10601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:16.433948 10653 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2c9e3ac96bc54ab2a397e3e94d422074 (127.3.217.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: NON_VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: true } } }
I20260924 07:06:16.435338 10790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: NON_VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: true } } }
I20260924 07:06:16.435441 10790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:06:16.439643 10608 master.cc:552] Master@127.3.217.60:38295 shutdown complete.
I20260924 07:06:16.439700 10608 master_runner.cc:305] Clearing existing system tablet
I20260924 07:06:16.439934 10608 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:16.439989 10608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:16.441161 10608 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:06:16.442019 10792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:16.442261 10608 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:16.442386 10608 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:06:16.442426 10608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/wal
uuid: "8571268ea3b24b0ca48031986cc19c2e"
format_stamp: "Formatted at 2026-09-24 07:06:16 on dist-test-slave-2p7j"
server_key: "c58fb3a5fe443691daaafc1e3acbe731"
server_key_iv: "4a4c4ef03e920e24b13e98e0b3f9c692"
server_key_version: "encryptionkey@0"
I20260924 07:06:16.444330 10608 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:06:16.446504 10608 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 07:06:16.446586 10608 log.cc:1196] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 07:06:16.446791 10608 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e: Deleting consensus metadata
I20260924 07:06:16.447059 10608 master_runner.cc:315] Copying system tablet from 127.3.217.62:36411
I20260924 07:06:16.448151 10608 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.217.62:36411
I20260924 07:06:16.460114 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.452017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59908 (local address 127.3.217.62:36411)
0924 07:06:16.452139 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:16.452143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:16.452264 (+ 121us) server_negotiation.cc:407] Connection header received
0924 07:06:16.452400 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:16.452404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:16.452454 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:16.452550 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:16.453048 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.453834 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.454950 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.455303 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.455397 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.456608 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:16.456612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:16.456615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:16.456640 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:16.458203 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.458747 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.458751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.458753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.458838 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.459128 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.459131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.459133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.459328 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:16.459395 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:16.459826 (+ 431us) server_negotiation.cc:299] Negotiation successful
0924 07:06:16.459948 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:06:16.460594 10798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.451860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.0.0.1:59908)
0924 07:06:16.452175 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:16.452188 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:16.452308 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:16.452572 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:16.452574 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:16.452806 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:16.452934 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.452941 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.453942 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.453945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.454840 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.454849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.455402 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.455406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.455496 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.456063 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:16.456077 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:16.456474 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:16.458320 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.458323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.458325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.458645 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.458940 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.458942 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.458944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.459036 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.459531 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:16.459534 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:16.459641 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 07:06:16.460304 (+ 663us) client_negotiation.cc:240] Negotiation successful
0924 07:06:16.460438 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":82,"threads_started":1}
I20260924 07:06:16.461162 10061 tablet_copy_service.cc:140] P 8c46e717f5ae4e6a9c5e6849154e5295: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8571268ea3b24b0ca48031986cc19c2e ({username='oryx', principal='oryx/127.3.217.60@KRBTEST.COM'} at 127.0.0.1:59908)
I20260924 07:06:16.461258 10061 tablet_copy_service.cc:161] P 8c46e717f5ae4e6a9c5e6849154e5295: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8571268ea3b24b0ca48031986cc19c2e at {username='oryx', principal='oryx/127.3.217.60@KRBTEST.COM'} at 127.0.0.1:59908: session id = 8571268ea3b24b0ca48031986cc19c2e-00000000000000000000000000000000
I20260924 07:06:16.462301 10061 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 07:06:16.463487 10608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 07:06:16.466073 10608 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e: tablet copy: Starting download of 0 data blocks...
I20260924 07:06:16.466243 10608 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e: tablet copy: Starting download of 1 WAL segments...
I20260924 07:06:16.467604 10608 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 07:06:16.469249 10061 tablet_copy_service.cc:342] P 8c46e717f5ae4e6a9c5e6849154e5295: Request end of tablet copy session 8571268ea3b24b0ca48031986cc19c2e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.217.60@KRBTEST.COM'} at 127.0.0.1:59908
I20260924 07:06:16.469322 10061 tablet_copy_service.cc:433] P 8c46e717f5ae4e6a9c5e6849154e5295: ending tablet copy session 8571268ea3b24b0ca48031986cc19c2e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8571268ea3b24b0ca48031986cc19c2e
I20260924 07:06:16.476435 10608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:16.478152 10799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:16.478164 10802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:16.478389 10800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:16.478518 10608 server_base.cc:1034] running on GCE node
I20260924 07:06:16.478852 10608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:16.479130 10608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:16.480265 10608 hybrid_clock.cc:648] HybridClock initialized: now 1790233576480244 us; error 38 us; skew 500 ppm
Sep 24 07:06:16 dist-test-slave-2p7j krb5kdc[9996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233576, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:16.481751 10608 init.cc:377] Logged in from keytab as oryx/127.3.217.60@KRBTEST.COM (short username oryx)
I20260924 07:06:16.482172 10608 webserver.cc:460] Webserver started at http://127.3.217.60:33189/ using document root <none> and password file <none>
I20260924 07:06:16.482336 10608 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:16.482374 10608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:16.483843 10608 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:06:16.485018 10809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:16.485283 10608 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:06:16.485358 10608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/wal
uuid: "8571268ea3b24b0ca48031986cc19c2e"
format_stamp: "Formatted at 2026-09-24 07:06:16 on dist-test-slave-2p7j"
server_key: "c58fb3a5fe443691daaafc1e3acbe731"
server_key_iv: "4a4c4ef03e920e24b13e98e0b3f9c692"
server_key_version: "encryptionkey@0"
I20260924 07:06:16.485432 10608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:16.502768 10608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:16.505004 10608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:16.509945 10608 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.60:38295
I20260924 07:06:16.509970 10861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.60:38295 every 8 connection(s)
I20260924 07:06:16.510715 10608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/master-2/data/info.pb
I20260924 07:06:16.512552 10862 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:06:16.513590 10862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e: Bootstrap starting.
I20260924 07:06:16.525393 10366 heartbeater.cc:503] Master 127.3.217.62:36411 was elected leader, sending a full tablet report...
I20260924 07:06:16.525630 10862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:06:16.525920 10862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e: Bootstrap complete.
I20260924 07:06:16.526445 10862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: NON_VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: true } }
I20260924 07:06:16.526572 10862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8571268ea3b24b0ca48031986cc19c2e, State: Initialized, Role: LEARNER
I20260924 07:06:16.526659 10862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: NON_VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: true } }
I20260924 07:06:16.527086 10862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:06:16.527352 10865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: NON_VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: true } } }
I20260924 07:06:16.527581 10865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [sys.catalog]: This master's current role is: LEARNER
I20260924 07:06:16.530622 10879 catalog_manager.cc:1259] Loaded cluster ID: 5730979573034234b8f25895f606f0bd
I20260924 07:06:16.530682 10879 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e: loading cluster ID for follower catalog manager: success
I20260924 07:06:16.533401 10879 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e: acquiring CA information for follower catalog manager: success
I20260924 07:06:16.534749 10879 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 07:06:16.535194 10788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:16.535403 10788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:16.535457 10788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:16.539485 10788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:16.539572 10788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:16.539651 10788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:06:16.544838 10788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:44339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.1.keytab
--principal=oryx/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:37477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=43471
--tserver_master_addrs=127.3.217.62:36411,127.3.217.61:44701,127.3.217.60:38295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:16.545779 10788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:16.546691 10788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:16.555840 10881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:16.555873 10884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:16.555922 10882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:16.556137 10788 server_base.cc:1034] running on GCE node
I20260924 07:06:16.556859 10788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:16.557621 10788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:16.558881 10788 hybrid_clock.cc:648] HybridClock initialized: now 1790233576558854 us; error 35 us; skew 500 ppm
Sep 24 07:06:16 dist-test-slave-2p7j krb5kdc[9996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233576, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:16.562161 10788 init.cc:377] Logged in from keytab as oryx/127.3.217.1@KRBTEST.COM (short username oryx)
I20260924 07:06:16.563278 10788 webserver.cc:460] Webserver started at http://127.3.217.1:43471/ using document root <none> and password file <none>
I20260924 07:06:16.564195 10788 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:16.564286 10788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:16.568506 10788 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260924 07:06:16.570598 10891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:16.571795 10788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:06:16.572230 10788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "921e678ee49a47b5a1be6be3174a856d"
format_stamp: "Formatted at 2026-09-24 07:06:15 on dist-test-slave-2p7j"
server_key: "611d537244c4350595fb96b06b56d481"
server_key_iv: "c4bf72eaa7a1465b431b5f98be721647"
server_key_version: "encryptionkey@0"
I20260924 07:06:16.572786 10788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:16.591653 10788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:16.595331 10788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:16.595556 10788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:16.596167 10788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:16.597059 10788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:16.597105 10788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:16.597157 10788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:16.597184 10788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:16.606468 10788 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:37477
I20260924 07:06:16.606489 11004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:37477 every 8 connection(s)
I20260924 07:06:16.607865 10788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:06:16.615953 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 10788
I20260924 07:06:16.616135 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 10237
Sep 24 07:06:16 dist-test-slave-2p7j krb5kdc[9996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233576, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM
Sep 24 07:06:16 dist-test-slave-2p7j krb5kdc[9996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233576, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.60@KRBTEST.COM
I20260924 07:06:16.622622 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:45971
--local_ip_for_outbound_sockets=127.3.217.2
--tserver_master_addrs=127.3.217.62:36411
--webserver_port=40975
--webserver_interface=127.3.217.2
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.2.keytab
--principal=oryx/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:44339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.217.62:36411,127.3.217.61:44701
--tserver_master_addrs=127.3.217.62:36411,127.3.217.61:44701,127.3.217.60:38295 with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:06:16.623703 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.611674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:58631 (local address 127.3.217.62:36411)
0924 07:06:16.611823 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:16.611827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:16.611933 (+ 106us) server_negotiation.cc:407] Connection header received
0924 07:06:16.612444 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:16.612447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:16.612493 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:16.612565 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:16.613922 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.614434 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.615906 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.616222 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.616299 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.620060 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:16.620064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:16.620066 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:16.620091 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:16.621450 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.622001 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.622005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.622007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.622078 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.622446 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.622450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.622451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.622654 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:16.622739 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:16.623397 (+ 658us) server_negotiation.cc:299] Negotiation successful
0924 07:06:16.623548 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:06:16.623726 11009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.609817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:51313 (local address 127.3.217.60:38295)
0924 07:06:16.610093 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:16.610097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:16.611303 (+ 1206us) server_negotiation.cc:407] Connection header received
0924 07:06:16.612677 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:16.612680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:16.612727 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:16.612807 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:16.613927 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.614903 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.615846 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.616182 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.617124 (+ 942us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.620334 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:16.620337 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:16.620339 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:16.620359 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:16.621773 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.622216 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.622219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.622221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.622294 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.622576 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.622579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.622580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.622777 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:16.622854 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:16.623391 (+ 537us) server_negotiation.cc:299] Negotiation successful
0924 07:06:16.623566 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":92,"threads_started":1}
I20260924 07:06:16.624696 11010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.610627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:38295 (local address 127.3.217.1:51313)
0924 07:06:16.611173 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:16.611213 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:16.612247 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:16.612843 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:16.612845 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:16.613201 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:16.613779 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.613785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.614993 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.614996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.615749 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.615757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.617128 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.617131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.617215 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.617550 (+ 335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:16.617562 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:16.619895 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:16.621899 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.621902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.621905 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.622127 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.622381 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.622384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.622386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.622475 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.622892 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:16.622894 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:16.623101 (+ 207us) client_negotiation.cc:769] Sending connection context
0924 07:06:16.623854 (+ 753us) client_negotiation.cc:240] Negotiation successful
0924 07:06:16.623979 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":164,"spinlock_wait_cycles":9472,"thread_start_us":129,"threads_started":1}
I20260924 07:06:16.624696 11011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.611574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.3.217.1:58631)
0924 07:06:16.611836 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:16.611848 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:16.612247 (+ 399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:16.612690 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:16.612696 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:16.613207 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:16.613776 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.613786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.614571 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.614573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.615802 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.615812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.616430 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.616433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.616596 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.617399 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:16.617414 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:16.619898 (+ 2484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:16.621576 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.621582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.621596 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.621907 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.622155 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.622159 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.622160 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.622324 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.622790 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:16.622796 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:16.623101 (+ 305us) client_negotiation.cc:769] Sending connection context
0924 07:06:16.623311 (+ 210us) client_negotiation.cc:240] Negotiation successful
0924 07:06:16.623657 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":107,"spinlock_wait_cycles":16128,"thread_start_us":104,"threads_started":1}
I20260924 07:06:16.625795 11006 heartbeater.cc:347] Connected to a master server at 127.3.217.62:36411
I20260924 07:06:16.625794 11005 heartbeater.cc:347] Connected to a master server at 127.3.217.60:38295
I20260924 07:06:16.626062 11005 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:16.626062 11006 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:16.626645 11006 heartbeater.cc:511] Master 127.3.217.62:36411 requested a full tablet report, sending...
I20260924 07:06:16.626645 11005 heartbeater.cc:511] Master 127.3.217.60:38295 requested a full tablet report, sending...
Sep 24 07:06:16 dist-test-slave-2p7j krb5kdc[9996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233576, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.61@KRBTEST.COM
I20260924 07:06:16.627866 10041 ts_manager.cc:194] Re-registered known tserver with Master: 921e678ee49a47b5a1be6be3174a856d (127.3.217.1:37477)
I20260924 07:06:16.628123 10825 ts_manager.cc:194] Registered new tserver with Master: 921e678ee49a47b5a1be6be3174a856d (127.3.217.1:37477)
I20260924 07:06:16.628587 10041 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.217.1@KRBTEST.COM'} at 127.3.217.1:58631
I20260924 07:06:16.631654 10765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.612045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:42521 (local address 127.3.217.61:44701)
0924 07:06:16.612184 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:16.612189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:16.612344 (+ 155us) server_negotiation.cc:407] Connection header received
0924 07:06:16.612468 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:16.612471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:16.612517 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:16.612565 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:16.613895 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.614682 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.624188 (+ 9506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.624524 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.624595 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.627816 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:16.627821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:16.627824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:16.627850 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:16.629391 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.630051 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.630053 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.630054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.630125 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.630596 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.630598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.630599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.630761 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:16.630842 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:16.631249 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 07:06:16.631343 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:06:16.631791 11012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.611963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:44701 (local address 127.3.217.1:42521)
0924 07:06:16.612256 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:16.612267 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:16.612358 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:16.612702 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:16.612705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:16.613198 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:16.613768 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.613781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.614789 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.614792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.624061 (+ 9269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.624071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.624761 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.624765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.624856 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.625882 (+ 1026us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:16.625896 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:16.627666 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:16.629511 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.629515 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.629516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.629746 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.630262 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.630266 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.630269 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.630449 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.630925 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:16.630930 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:16.631046 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 07:06:16.631180 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 07:06:16.631340 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":8351,"thread_start_us":108,"threads_started":1}
I20260924 07:06:16.632445 11007 heartbeater.cc:347] Connected to a master server at 127.3.217.61:44701
I20260924 07:06:16.632537 11007 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:16.632738 11007 heartbeater.cc:511] Master 127.3.217.61:44701 requested a full tablet report, sending...
I20260924 07:06:16.633808 10643 ts_manager.cc:194] Registered new tserver with Master: 921e678ee49a47b5a1be6be3174a856d (127.3.217.1:37477)
W20260924 07:06:16.736711 11013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:16.736944 11013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:16.736984 11013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:16.740999 11013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:16.741058 11013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:16.741114 11013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:06:16.746451 11013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:44339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.2.keytab
--principal=oryx/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:45971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=40975
--tserver_master_addrs=127.3.217.62:36411,127.3.217.61:44701,127.3.217.60:38295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:16.747627 11013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:16.748764 11013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:06:16.752374 10500 heartbeater.cc:503] Master 127.3.217.62:36411 was elected leader, sending a full tablet report...
W20260924 07:06:16.758466 11018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:16.758562 11021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:16.758889 11013 server_base.cc:1034] running on GCE node
W20260924 07:06:16.758405 11019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:16.759713 11013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:16.760349 11013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:16.761538 11013 hybrid_clock.cc:648] HybridClock initialized: now 1790233576761522 us; error 27 us; skew 500 ppm
Sep 24 07:06:16 dist-test-slave-2p7j krb5kdc[9996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233576, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:16.764132 11013 init.cc:377] Logged in from keytab as oryx/127.3.217.2@KRBTEST.COM (short username oryx)
I20260924 07:06:16.764900 11013 webserver.cc:460] Webserver started at http://127.3.217.2:40975/ using document root <none> and password file <none>
I20260924 07:06:16.765384 11013 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:16.765448 11013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:16.769521 11013 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 07:06:16.771559 11028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:16.773002 11013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260924 07:06:16.773115 11013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "6eafe362852c419ba36a9014bffa7f03"
format_stamp: "Formatted at 2026-09-24 07:06:15 on dist-test-slave-2p7j"
server_key: "492aebc166d308e24a33d5904efe7b45"
server_key_iv: "8d76d2f63ae32548082f6f9bd261ff9f"
server_key_version: "encryptionkey@0"
I20260924 07:06:16.773627 11013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:16.790311 11013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:16.793948 11013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:16.794175 11013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:16.794735 11013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:16.795610 11013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:16.795656 11013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:16.795694 11013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:16.795708 11013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:16.805120 11013 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:45971
I20260924 07:06:16.805154 11141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:45971 every 8 connection(s)
I20260924 07:06:16.806150 11013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:06:16.810724 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 11013
I20260924 07:06:16.810946 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 10371
I20260924 07:06:16.820175 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:36493
--local_ip_for_outbound_sockets=127.3.217.3
--tserver_master_addrs=127.3.217.62:36411
--webserver_port=41561
--webserver_interface=127.3.217.3
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.3.keytab
--principal=oryx/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:44339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.217.62:36411,127.3.217.61:44701
--tserver_master_addrs=127.3.217.62:36411,127.3.217.61:44701,127.3.217.60:38295 with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:06:16 dist-test-slave-2p7j krb5kdc[9996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233576, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM
Sep 24 07:06:16 dist-test-slave-2p7j krb5kdc[9996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233576, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.60@KRBTEST.COM
Sep 24 07:06:16 dist-test-slave-2p7j krb5kdc[9996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233576, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.61@KRBTEST.COM
I20260924 07:06:16.826529 11009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.809809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:48473 (local address 127.3.217.60:38295)
0924 07:06:16.809957 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:16.809962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:16.813410 (+ 3448us) server_negotiation.cc:407] Connection header received
0924 07:06:16.814313 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:16.814317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:16.814373 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:16.814424 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:16.816439 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.817342 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.818664 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.819023 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.819102 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.822598 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:16.822601 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:16.822604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:16.822632 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:16.824341 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.824915 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.824919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.824920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.825000 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.825356 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.825359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.825361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.825570 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:16.825646 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:16.826247 (+ 601us) server_negotiation.cc:299] Negotiation successful
0924 07:06:16.826376 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 07:06:16.826529 10765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.808591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:46869 (local address 127.3.217.61:44701)
0924 07:06:16.808721 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:16.808726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:16.810560 (+ 1834us) server_negotiation.cc:407] Connection header received
0924 07:06:16.814337 (+ 3777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:16.814342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:16.814408 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:16.814463 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:16.817543 (+ 3080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.818274 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.819677 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.819995 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.820083 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.823389 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:16.823393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:16.823396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:16.823422 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:16.824907 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.825326 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.825330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.825332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.825408 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.825674 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.825678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.825680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.825895 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:16.825972 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:16.826252 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 07:06:16.826362 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:06:16.826828 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.808622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:37289 (local address 127.3.217.62:36411)
0924 07:06:16.808725 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:16.808729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:16.812939 (+ 4210us) server_negotiation.cc:407] Connection header received
0924 07:06:16.814310 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:16.814313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:16.814372 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:16.814424 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:16.816492 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.817011 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.818663 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.819042 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.819103 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.823046 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:16.823050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:16.823052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:16.823077 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:16.825049 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.825511 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.825514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.825516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.825597 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:16.825884 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:16.825901 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:16.825903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:16.826121 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:16.826201 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:16.826545 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 07:06:16.826673 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:06:16.828276 11147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.809294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:44701 (local address 127.3.217.2:46869)
0924 07:06:16.810355 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:16.810420 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:16.813992 (+ 3572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:16.814858 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:16.814867 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:16.815746 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:16.816243 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.816262 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.818435 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.818439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.819496 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.819533 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.820282 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.820287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.820388 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.820875 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:16.820890 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:16.823230 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:16.825009 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.825012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.825014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.825221 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.825505 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.825508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.825509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.825583 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.826025 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:16.826028 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:16.826099 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 07:06:16.826805 (+ 706us) client_negotiation.cc:240] Negotiation successful
0924 07:06:16.827174 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"mutex_wait_us":203,"thread_start_us":148,"threads_started":1}
I20260924 07:06:16.828277 11148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.809271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.3.217.2:37289)
0924 07:06:16.812813 (+ 3542us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:16.812831 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:16.814026 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:16.814857 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:16.814867 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:16.815417 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:16.816389 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.816395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.817141 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.817144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.818526 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.818537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.819298 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.819302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.819592 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.820238 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:16.820262 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:16.822918 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:16.825173 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.825176 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.825179 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.825401 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.825698 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.825701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.825703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.825783 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.826258 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:16.826261 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:16.826360 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 07:06:16.827429 (+ 1069us) client_negotiation.cc:240] Negotiation successful
0924 07:06:16.827600 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3432,"mutex_wait_us":706,"spinlock_wait_cycles":6400,"thread_start_us":221,"threads_started":1}
I20260924 07:06:16.829016 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:16.809595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:38295 (local address 127.3.217.2:48473)
0924 07:06:16.813206 (+ 3611us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:16.813226 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:16.814019 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:16.814857 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:16.814867 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:16.815416 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:16.816243 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.816262 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.817494 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.817497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.818526 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:16.818537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:16.819298 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:16.819302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:16.819584 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:16.820563 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:16.820577 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:16.822438 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:16.824470 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.824476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.824492 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.824802 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.825095 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:16.825097 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:16.825099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:16.825247 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:16.825704 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:16.825710 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:16.825990 (+ 280us) client_negotiation.cc:769] Sending connection context
0924 07:06:16.827154 (+ 1164us) client_negotiation.cc:240] Negotiation successful
0924 07:06:16.827260 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3461,"mutex_wait_us":185,"spinlock_wait_cycles":4224,"thread_start_us":111,"threads_started":1}
I20260924 07:06:16.829581 11143 heartbeater.cc:347] Connected to a master server at 127.3.217.62:36411
I20260924 07:06:16.829825 11143 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:16.829967 11142 heartbeater.cc:347] Connected to a master server at 127.3.217.60:38295
I20260924 07:06:16.830039 11142 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:16.830237 11144 heartbeater.cc:347] Connected to a master server at 127.3.217.61:44701
I20260924 07:06:16.830302 11144 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:16.830421 11144 heartbeater.cc:511] Master 127.3.217.61:44701 requested a full tablet report, sending...
I20260924 07:06:16.830406 11143 heartbeater.cc:511] Master 127.3.217.62:36411 requested a full tablet report, sending...
I20260924 07:06:16.830405 11142 heartbeater.cc:511] Master 127.3.217.60:38295 requested a full tablet report, sending...
I20260924 07:06:16.831543 10041 ts_manager.cc:194] Re-registered known tserver with Master: 6eafe362852c419ba36a9014bffa7f03 (127.3.217.2:45971)
I20260924 07:06:16.831549 10825 ts_manager.cc:194] Registered new tserver with Master: 6eafe362852c419ba36a9014bffa7f03 (127.3.217.2:45971)
I20260924 07:06:16.831549 10643 ts_manager.cc:194] Registered new tserver with Master: 6eafe362852c419ba36a9014bffa7f03 (127.3.217.2:45971)
I20260924 07:06:16.832278 10041 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.217.2@KRBTEST.COM'} at 127.3.217.2:37289
W20260924 07:06:16.934643 11149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:16.934875 11149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:16.934940 11149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:16.940089 11149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:16.940212 11149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:16.940321 11149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:06:16.947759 11149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:44339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.3.keytab
--principal=oryx/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:36493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=41561
--tserver_master_addrs=127.3.217.62:36411,127.3.217.61:44701,127.3.217.60:38295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:16.948778 11149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:16.949738 11149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:16.960621 11157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:16.960788 11149 server_base.cc:1034] running on GCE node
W20260924 07:06:16.960613 11154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:16.960608 11155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:16.961716 11149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:16.962445 11149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:16.963639 11149 hybrid_clock.cc:648] HybridClock initialized: now 1790233576963578 us; error 74 us; skew 500 ppm
Sep 24 07:06:16 dist-test-slave-2p7j krb5kdc[9996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233576, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:16.966542 11149 init.cc:377] Logged in from keytab as oryx/127.3.217.3@KRBTEST.COM (short username oryx)
I20260924 07:06:16.967382 11149 webserver.cc:460] Webserver started at http://127.3.217.3:41561/ using document root <none> and password file <none>
I20260924 07:06:16.967919 11149 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:16.967996 11149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:16.971997 11149 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 07:06:16.973997 11164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:16.975817 11149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:06:16.976084 11149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4"
format_stamp: "Formatted at 2026-09-24 07:06:15 on dist-test-slave-2p7j"
server_key: "18368c47c9c46c0bcaa3a93e3fd529d4"
server_key_iv: "3a93109d106a73400669ae2b95b08ef3"
server_key_version: "encryptionkey@0"
I20260924 07:06:16.977031 11149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:17.010700 11149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:17.013964 11149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:17.014128 11149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:17.014680 11149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:17.015549 11149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:17.015595 11149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:17.015646 11149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:17.015672 11149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:17.026173 11149 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:36493
I20260924 07:06:17.026232 11278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:36493 every 8 connection(s)
I20260924 07:06:17.027145 11149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:06:17.027774 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 11149
Sep 24 07:06:17 dist-test-slave-2p7j krb5kdc[9996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233576, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.60@KRBTEST.COM
Sep 24 07:06:17 dist-test-slave-2p7j krb5kdc[9996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233576, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM
Sep 24 07:06:17 dist-test-slave-2p7j krb5kdc[9996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233576, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.61@KRBTEST.COM
I20260924 07:06:17.041728 10765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.029993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47670 (local address 127.3.217.61:44701)
0924 07:06:17.030100 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:17.030105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:17.031646 (+ 1541us) server_negotiation.cc:407] Connection header received
0924 07:06:17.031855 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:17.031859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:17.031909 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:17.031960 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:17.032915 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.033728 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.034845 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.035180 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.035251 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.038391 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:17.038395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:17.038398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:17.038420 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:17.039997 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:17.040420 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:17.040423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:17.040425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:17.040501 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:17.040802 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:17.040805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:17.040807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:17.041016 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:17.041115 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:17.041453 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 07:06:17.041558 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:06:17.043848 11009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.031040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:41249 (local address 127.3.217.60:38295)
0924 07:06:17.031360 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:17.031364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:17.031381 (+ 17us) server_negotiation.cc:407] Connection header received
0924 07:06:17.031424 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:17.031427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:17.031473 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:17.031579 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:17.033199 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.033939 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.035687 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.036019 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.036082 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.039748 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:17.039752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:17.039755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:17.039778 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:17.041297 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:17.041869 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:17.041873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:17.041874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:17.041949 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:17.042304 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:17.042307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:17.042309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:17.042496 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:17.042733 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:17.043585 (+ 852us) server_negotiation.cc:299] Negotiation successful
0924 07:06:17.043704 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260924 07:06:17.045550 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.029361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:38295 (local address 127.3.217.3:41249)
0924 07:06:17.030132 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:17.030187 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:17.031185 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:17.031905 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:17.031913 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:17.032369 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:17.033035 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:17.033051 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.034056 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.034059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:17.035570 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:17.035579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.036244 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.036248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:17.036456 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.036950 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:17.036969 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:17.039579 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:17.041414 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:17.041419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:17.041432 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:17.041746 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:17.042052 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:17.042054 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:17.042056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:17.042212 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:17.042822 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:17.042836 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:17.043210 (+ 374us) client_negotiation.cc:769] Sending connection context
0924 07:06:17.044015 (+ 805us) client_negotiation.cc:240] Negotiation successful
0924 07:06:17.044332 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":187,"spinlock_wait_cycles":1920,"thread_start_us":125,"threads_started":1}
I20260924 07:06:17.046787 11279 heartbeater.cc:347] Connected to a master server at 127.3.217.60:38295
I20260924 07:06:17.047008 11279 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:17.047459 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.037762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:38295 (local address 127.0.0.1:52890)
0924 07:06:17.038823 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:17.038835 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:17.039252 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:17.043195 (+ 3943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:17.043198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:17.043215 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:17.043293 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:17.043298 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.044581 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.044586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:17.045960 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:17.045969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.047075 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.047079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:17.047172 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.047193 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:06:17.047253 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 07:06:17.047325 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"thread_start_us":62,"threads_started":1}
I20260924 07:06:17.047858 11279 heartbeater.cc:511] Master 127.3.217.60:38295 requested a full tablet report, sending...
I20260924 07:06:17.048035 11297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.038039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.3.217.60:38295)
0924 07:06:17.038316 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:17.038321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:17.039189 (+ 868us) server_negotiation.cc:407] Connection header received
0924 07:06:17.042949 (+ 3760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:17.042952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:17.043013 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:17.043105 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:17.043391 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.044317 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.046105 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.046773 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.046872 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.047013 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:17.047671 (+ 658us) server_negotiation.cc:299] Negotiation successful
0924 07:06:17.047742 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
I20260924 07:06:17.048254 11292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.030820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59912 (local address 127.3.217.62:36411)
0924 07:06:17.031804 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:17.031813 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:17.031825 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:06:17.031865 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:17.031868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:17.031912 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:17.035285 (+ 3373us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:17.035401 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.036136 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.038747 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.039063 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.041068 (+ 2005us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.042087 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:17.042091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:17.042093 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:17.042117 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:17.045395 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:17.045973 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:17.045975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:17.045977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:17.046060 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:17.047186 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:17.047189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:17.047190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:17.047370 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:17.047938 (+ 568us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:17.048010 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 07:06:17.048120 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":927,"thread_start_us":74,"threads_started":1}
I20260924 07:06:17.048514 10836 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 07:06:17.049185 10825 ts_manager.cc:194] Registered new tserver with Master: 25fe6ef8b01840ecb7cfae4d871a0ba4 (127.3.217.3:36493)
I20260924 07:06:17.049525 11298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c46e717f5ae4e6a9c5e6849154e5295. Latest consensus state: current_term: 1 leader_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: NON_VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: true } } }
I20260924 07:06:17.049646 11298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [sys.catalog]: This master's current role is: LEARNER
I20260924 07:06:17.050184 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.030737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:51271 (local address 127.3.217.62:36411)
0924 07:06:17.031661 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:17.031665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:17.031944 (+ 279us) server_negotiation.cc:407] Connection header received
0924 07:06:17.032039 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:17.032042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:17.032075 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:17.032136 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:17.033691 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.034392 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.036550 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.036838 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.036906 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.039979 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:17.039983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:17.039985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:17.040008 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:17.042053 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:17.043648 (+ 1595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:17.043651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:17.043653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:17.043735 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:17.045495 (+ 1760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:17.045498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:17.045500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:17.045701 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:17.045781 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:17.050008 (+ 4227us) server_negotiation.cc:299] Negotiation successful
0924 07:06:17.050094 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831}
I20260924 07:06:17.050235 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.029361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.3.217.3:51271)
0924 07:06:17.031880 (+ 2519us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:17.031892 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:17.031975 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:17.032262 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:17.032265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:17.032434 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:17.033579 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:17.033585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.034512 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.034514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:17.036436 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:17.036444 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.037053 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.037057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:17.037148 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.037523 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:17.037537 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:17.039850 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:17.042703 (+ 2853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:17.042706 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:17.042708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:17.042962 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:17.043839 (+ 877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:17.043841 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:17.043843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:17.044943 (+ 1100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:17.046355 (+ 1412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:17.046357 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:17.046454 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 07:06:17.049918 (+ 3464us) client_negotiation.cc:240] Negotiation successful
0924 07:06:17.050049 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2443,"mutex_wait_us":1911,"spinlock_wait_cycles":31744,"thread_start_us":122,"threads_started":1}
I20260924 07:06:17.050734 11280 heartbeater.cc:347] Connected to a master server at 127.3.217.62:36411
I20260924 07:06:17.050823 11280 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:17.050983 11280 heartbeater.cc:511] Master 127.3.217.62:36411 requested a full tablet report, sending...
I20260924 07:06:17.051594 10039 ts_manager.cc:194] Re-registered known tserver with Master: 25fe6ef8b01840ecb7cfae4d871a0ba4 (127.3.217.3:36493)
I20260924 07:06:17.052145 11295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.031411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52884 (local address 127.3.217.60:38295)
0924 07:06:17.036329 (+ 4918us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:17.036333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:17.036922 (+ 589us) server_negotiation.cc:407] Connection header received
0924 07:06:17.036961 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:17.036964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:17.037007 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:17.037673 (+ 666us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:17.039943 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.040652 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.043175 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.043495 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.043612 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.046971 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:17.046974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:17.046977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:17.047000 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:17.050168 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:17.050617 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:17.050620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:17.050622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:17.050700 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:17.050940 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:17.050944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:17.050945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:17.051153 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:17.051270 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:17.051929 (+ 659us) server_negotiation.cc:299] Negotiation successful
0924 07:06:17.052039 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4842,"thread_start_us":96,"threads_started":1}
I20260924 07:06:17.052232 10039 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.217.3@KRBTEST.COM'} at 127.3.217.3:51271
I20260924 07:06:17.051250 10041 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:06:17.054996 10041 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:06:17.056219 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.031297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:50489 (local address 127.3.217.61:44701)
0924 07:06:17.032423 (+ 1126us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:17.032427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:17.032439 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:06:17.032468 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:17.032471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:17.032524 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:17.032601 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:17.033171 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.033867 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.035689 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.036056 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.036146 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.043179 (+ 7033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:17.043183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:17.043185 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:17.043206 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:17.054429 (+ 11223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:17.054840 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:17.054843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:17.054845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:17.054930 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:17.055348 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:17.055352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:17.055354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:17.055595 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:17.055687 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:17.055941 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 07:06:17.056054 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1062,"thread_start_us":79,"threads_started":1}
I20260924 07:06:17.056478 11293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.030980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:44701 (local address 127.3.217.3:50489)
0924 07:06:17.032169 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:17.032181 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:17.032254 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:17.032705 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:17.032708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:17.032871 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:17.033034 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:17.033051 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.033998 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.034001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:17.035566 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:17.035575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.036249 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.036253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:17.036741 (+ 488us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.037243 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:17.037256 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:17.039670 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:17.054532 (+ 14862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:17.054535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:17.054536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:17.054722 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:17.055039 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:17.055041 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:17.055043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:17.055152 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:17.055680 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:17.055683 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:17.055766 (+ 83us) client_negotiation.cc:769] Sending connection context
0924 07:06:17.055849 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:06:17.055973 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1134,"mutex_wait_us":463,"thread_start_us":111,"threads_started":1}
I20260924 07:06:17.057322 11281 heartbeater.cc:347] Connected to a master server at 127.3.217.61:44701
I20260924 07:06:17.057471 11281 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:17.057699 11281 heartbeater.cc:511] Master 127.3.217.61:44701 requested a full tablet report, sending...
I20260924 07:06:17.057809 11300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: NON_VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: true } } }
I20260924 07:06:17.057914 11300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:17.060159 10643 ts_manager.cc:194] Registered new tserver with Master: 25fe6ef8b01840ecb7cfae4d871a0ba4 (127.3.217.3:36493)
I20260924 07:06:17.061528 11301 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295: attempting to promote NON_VOTER 8571268ea3b24b0ca48031986cc19c2e to VOTER
I20260924 07:06:17.061940 11301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: false } }
I20260924 07:06:17.063036 10653 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [term 1 FOLLOWER]: Refusing update from remote peer 8c46e717f5ae4e6a9c5e6849154e5295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 07:06:17.063429 11301 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 07:06:17.063910 10835 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [term 1 LEARNER]: Refusing update from remote peer 8c46e717f5ae4e6a9c5e6849154e5295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 07:06:17.064457 11301 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 07:06:17.065465 11300 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8571268ea3b24b0ca48031986cc19c2e (127.3.217.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: false } } }
I20260924 07:06:17.066864 11300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: false } } }
I20260924 07:06:17.066978 11300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:17.067016 11301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: false } } }
I20260924 07:06:17.067113 11301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:17.067092 10604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: false } } }
I20260924 07:06:17.067167 10604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:17.067137 10835 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8571268ea3b24b0ca48031986cc19c2e (127.3.217.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: false } } }
I20260924 07:06:17.067817 10653 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8571268ea3b24b0ca48031986cc19c2e (127.3.217.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: false } } }
I20260924 07:06:17.068717 11298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: false } } }
I20260924 07:06:17.068831 11298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:06:17.069547 11310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: false } } }
I20260924 07:06:17.069722 11310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:06:17.077093 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.071239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:45971 (local address 127.0.0.1:49406)
0924 07:06:17.071383 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:17.071396 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:17.071458 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:17.072986 (+ 1528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:17.072989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:17.073005 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:17.073136 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:17.073143 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.074118 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.074122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:17.075031 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:17.075040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.076612 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.076616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:17.076729 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.076766 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 07:06:17.076819 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 07:06:17.076902 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":25}
I20260924 07:06:17.077963 11312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.071761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:36493 (local address 127.0.0.1:44214)
0924 07:06:17.071985 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:17.071998 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:17.072087 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:17.073019 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:17.073023 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:17.073052 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:17.073175 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:17.073181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.074622 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.074625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:17.075849 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:17.075864 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.077550 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.077554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:17.077646 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.077682 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 07:06:17.077735 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 07:06:17.077807 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"spinlock_wait_cycles":8448,"thread_start_us":90,"threads_started":1}
I20260924 07:06:17.078122 11311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.071669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49406 (local address 127.3.217.2:45971)
0924 07:06:17.072634 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:17.072643 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:17.072661 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:06:17.072719 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:17.072723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:17.072869 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:17.072998 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:17.073235 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.074023 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.075192 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.076417 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.077493 (+ 1076us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.077766 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:17.077824 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 07:06:17.077935 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":869,"thread_start_us":126,"threads_started":1}
I20260924 07:06:17.078645 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.072142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44214 (local address 127.3.217.3:36493)
0924 07:06:17.072498 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:17.072512 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:17.072535 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:06:17.072609 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:17.072614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:17.072826 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:17.073304 (+ 478us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:17.073402 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.074504 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.076071 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.077277 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.078231 (+ 954us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.078396 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:17.078446 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 07:06:17.078535 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":87,"threads_started":1}
I20260924 07:06:17.079829 11313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.072093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:37477 (local address 127.0.0.1:53686)
0924 07:06:17.072294 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:17.072306 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:17.072388 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:17.073240 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:17.073242 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:17.073258 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:17.073337 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:17.073343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.075723 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.075729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:17.077491 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:17.077502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.079411 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.079415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:17.079534 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.079558 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 07:06:17.079624 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 07:06:17.079703 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
I20260924 07:06:17.080402 11315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.072440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53686 (local address 127.3.217.1:37477)
0924 07:06:17.072752 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:17.072764 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:17.072783 (+ 19us) server_negotiation.cc:407] Connection header received
0924 07:06:17.072849 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:17.072854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:17.073013 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:17.073144 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:17.073440 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.074516 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.077774 (+ 3258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.079302 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.079412 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.079598 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:17.080177 (+ 579us) server_negotiation.cc:299] Negotiation successful
0924 07:06:17.080277 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":85,"threads_started":1}
I20260924 07:06:17.080946 11076 tablet_service.cc:1467] Processing CreateTablet for tablet e3db2dfa9c184f498ef8d9af278d61c6 (DEFAULT_TABLE table=test-table [id=0bfac6a121524aa69fcfec3a0f9cff0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:17.081140 11212 tablet_service.cc:1467] Processing CreateTablet for tablet e3db2dfa9c184f498ef8d9af278d61c6 (DEFAULT_TABLE table=test-table [id=0bfac6a121524aa69fcfec3a0f9cff0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:17.082026 11076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3db2dfa9c184f498ef8d9af278d61c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:17.082026 11212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3db2dfa9c184f498ef8d9af278d61c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:17.082885 10939 tablet_service.cc:1467] Processing CreateTablet for tablet e3db2dfa9c184f498ef8d9af278d61c6 (DEFAULT_TABLE table=test-table [id=0bfac6a121524aa69fcfec3a0f9cff0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:17.083714 10939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3db2dfa9c184f498ef8d9af278d61c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:17.089556 11316 tablet_bootstrap.cc:492] T e3db2dfa9c184f498ef8d9af278d61c6 P 921e678ee49a47b5a1be6be3174a856d: Bootstrap starting.
I20260924 07:06:17.090448 11317 tablet_bootstrap.cc:492] T e3db2dfa9c184f498ef8d9af278d61c6 P 6eafe362852c419ba36a9014bffa7f03: Bootstrap starting.
I20260924 07:06:17.090724 11319 tablet_bootstrap.cc:492] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4: Bootstrap starting.
I20260924 07:06:17.091903 11316 tablet_bootstrap.cc:654] T e3db2dfa9c184f498ef8d9af278d61c6 P 921e678ee49a47b5a1be6be3174a856d: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:17.092553 11319 tablet_bootstrap.cc:654] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:17.092553 11317 tablet_bootstrap.cc:654] T e3db2dfa9c184f498ef8d9af278d61c6 P 6eafe362852c419ba36a9014bffa7f03: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:17.092772 11316 log.cc:824] T e3db2dfa9c184f498ef8d9af278d61c6 P 921e678ee49a47b5a1be6be3174a856d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:17.093303 11319 log.cc:824] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:17.093309 11317 log.cc:824] T e3db2dfa9c184f498ef8d9af278d61c6 P 6eafe362852c419ba36a9014bffa7f03: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:17.094599 11316 tablet_bootstrap.cc:492] T e3db2dfa9c184f498ef8d9af278d61c6 P 921e678ee49a47b5a1be6be3174a856d: No bootstrap required, opened a new log
I20260924 07:06:17.094771 11316 ts_tablet_manager.cc:1397] T e3db2dfa9c184f498ef8d9af278d61c6 P 921e678ee49a47b5a1be6be3174a856d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 07:06:17.094985 11317 tablet_bootstrap.cc:492] T e3db2dfa9c184f498ef8d9af278d61c6 P 6eafe362852c419ba36a9014bffa7f03: No bootstrap required, opened a new log
I20260924 07:06:17.095155 11317 ts_tablet_manager.cc:1397] T e3db2dfa9c184f498ef8d9af278d61c6 P 6eafe362852c419ba36a9014bffa7f03: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 07:06:17.095444 11319 tablet_bootstrap.cc:492] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4: No bootstrap required, opened a new log
I20260924 07:06:17.095633 11319 ts_tablet_manager.cc:1397] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260924 07:06:17.098409 11316 raft_consensus.cc:348] T e3db2dfa9c184f498ef8d9af278d61c6 P 921e678ee49a47b5a1be6be3174a856d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafe362852c419ba36a9014bffa7f03" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 45971 } } peers { permanent_uuid: "921e678ee49a47b5a1be6be3174a856d" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 37477 } } peers { permanent_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36493 } }
I20260924 07:06:17.098608 11316 raft_consensus.cc:374] T e3db2dfa9c184f498ef8d9af278d61c6 P 921e678ee49a47b5a1be6be3174a856d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:17.098658 11316 raft_consensus.cc:729] T e3db2dfa9c184f498ef8d9af278d61c6 P 921e678ee49a47b5a1be6be3174a856d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 921e678ee49a47b5a1be6be3174a856d, State: Initialized, Role: FOLLOWER
I20260924 07:06:17.099046 11316 consensus_queue.cc:260] T e3db2dfa9c184f498ef8d9af278d61c6 P 921e678ee49a47b5a1be6be3174a856d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafe362852c419ba36a9014bffa7f03" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 45971 } } peers { permanent_uuid: "921e678ee49a47b5a1be6be3174a856d" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 37477 } } peers { permanent_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36493 } }
I20260924 07:06:17.099478 11317 raft_consensus.cc:348] T e3db2dfa9c184f498ef8d9af278d61c6 P 6eafe362852c419ba36a9014bffa7f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafe362852c419ba36a9014bffa7f03" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 45971 } } peers { permanent_uuid: "921e678ee49a47b5a1be6be3174a856d" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 37477 } } peers { permanent_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36493 } }
I20260924 07:06:17.099727 11317 raft_consensus.cc:374] T e3db2dfa9c184f498ef8d9af278d61c6 P 6eafe362852c419ba36a9014bffa7f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:17.099781 11317 raft_consensus.cc:729] T e3db2dfa9c184f498ef8d9af278d61c6 P 6eafe362852c419ba36a9014bffa7f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eafe362852c419ba36a9014bffa7f03, State: Initialized, Role: FOLLOWER
I20260924 07:06:17.099967 11316 ts_tablet_manager.cc:1428] T e3db2dfa9c184f498ef8d9af278d61c6 P 921e678ee49a47b5a1be6be3174a856d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 07:06:17.100167 11317 consensus_queue.cc:260] T e3db2dfa9c184f498ef8d9af278d61c6 P 6eafe362852c419ba36a9014bffa7f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafe362852c419ba36a9014bffa7f03" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 45971 } } peers { permanent_uuid: "921e678ee49a47b5a1be6be3174a856d" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 37477 } } peers { permanent_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36493 } }
I20260924 07:06:17.100409 11006 heartbeater.cc:503] Master 127.3.217.62:36411 was elected leader, sending a full tablet report...
I20260924 07:06:17.100996 11317 ts_tablet_manager.cc:1428] T e3db2dfa9c184f498ef8d9af278d61c6 P 6eafe362852c419ba36a9014bffa7f03: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260924 07:06:17.100833 11319 raft_consensus.cc:348] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafe362852c419ba36a9014bffa7f03" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 45971 } } peers { permanent_uuid: "921e678ee49a47b5a1be6be3174a856d" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 37477 } } peers { permanent_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36493 } }
I20260924 07:06:17.101066 11319 raft_consensus.cc:374] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:17.101195 11319 raft_consensus.cc:729] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25fe6ef8b01840ecb7cfae4d871a0ba4, State: Initialized, Role: FOLLOWER
I20260924 07:06:17.101320 11143 heartbeater.cc:503] Master 127.3.217.62:36411 was elected leader, sending a full tablet report...
I20260924 07:06:17.101615 11319 consensus_queue.cc:260] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafe362852c419ba36a9014bffa7f03" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 45971 } } peers { permanent_uuid: "921e678ee49a47b5a1be6be3174a856d" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 37477 } } peers { permanent_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36493 } }
I20260924 07:06:17.102531 11319 ts_tablet_manager.cc:1428] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260924 07:06:17.104480 11280 heartbeater.cc:503] Master 127.3.217.62:36411 was elected leader, sending a full tablet report...
W20260924 07:06:17.109997 11008 tablet.cc:2369] T e3db2dfa9c184f498ef8d9af278d61c6 P 921e678ee49a47b5a1be6be3174a856d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:06:17.179725 11325 raft_consensus.cc:482] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:06:17.179924 11325 raft_consensus.cc:504] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafe362852c419ba36a9014bffa7f03" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 45971 } } peers { permanent_uuid: "921e678ee49a47b5a1be6be3174a856d" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 37477 } } peers { permanent_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36493 } }
I20260924 07:06:17.181808 11325 leader_election.cc:288] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6eafe362852c419ba36a9014bffa7f03 (127.3.217.2:45971), 921e678ee49a47b5a1be6be3174a856d (127.3.217.1:37477)
I20260924 07:06:17.185832 11293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.182081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:37477 (local address 127.3.217.3:52233)
0924 07:06:17.182303 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:17.182330 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:17.182527 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:17.182805 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:17.182808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:17.182824 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:17.182967 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:17.182973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.183718 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.183722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:17.184741 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:17.184753 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.185401 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.185404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:17.185508 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.185534 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 07:06:17.185617 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:06:17.185698 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":22,"spinlock_wait_cycles":7424}
I20260924 07:06:17.185878 11315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.182188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:52233 (local address 127.3.217.1:37477)
0924 07:06:17.182405 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:17.182409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:17.182431 (+ 22us) server_negotiation.cc:407] Connection header received
0924 07:06:17.182666 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:17.182669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:17.182710 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:17.182779 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:17.183079 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.183612 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.184888 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.185283 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.185360 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.185413 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:17.185667 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 07:06:17.185760 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260924 07:06:17.186087 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.182086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:45971 (local address 127.3.217.3:43077)
0924 07:06:17.182374 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:17.182388 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:17.182514 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:17.182810 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:17.182813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:17.182835 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:17.182975 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:17.182981 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.183878 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.183880 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:17.184734 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:17.184743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.185674 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.185678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:17.185781 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.185804 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:06:17.185883 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 07:06:17.185969 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":182,"spinlock_wait_cycles":17920}
I20260924 07:06:17.186100 11311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.182176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:43077 (local address 127.3.217.2:45971)
0924 07:06:17.182287 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:17.182291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:17.182486 (+ 195us) server_negotiation.cc:407] Connection header received
0924 07:06:17.182640 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:17.182644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:17.182702 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:17.182770 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:17.183105 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.183760 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.184881 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.185543 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.185640 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.185697 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:17.185912 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 07:06:17.185989 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 07:06:17.186635 10959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3db2dfa9c184f498ef8d9af278d61c6" candidate_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "921e678ee49a47b5a1be6be3174a856d" is_pre_election: true
I20260924 07:06:17.186646 11096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3db2dfa9c184f498ef8d9af278d61c6" candidate_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6eafe362852c419ba36a9014bffa7f03" is_pre_election: true
I20260924 07:06:17.186939 10959 raft_consensus.cc:2454] T e3db2dfa9c184f498ef8d9af278d61c6 P 921e678ee49a47b5a1be6be3174a856d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25fe6ef8b01840ecb7cfae4d871a0ba4 in term 0.
I20260924 07:06:17.186939 11096 raft_consensus.cc:2454] T e3db2dfa9c184f498ef8d9af278d61c6 P 6eafe362852c419ba36a9014bffa7f03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25fe6ef8b01840ecb7cfae4d871a0ba4 in term 0.
I20260924 07:06:17.187403 11167 leader_election.cc:302] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 25fe6ef8b01840ecb7cfae4d871a0ba4, 6eafe362852c419ba36a9014bffa7f03, 921e678ee49a47b5a1be6be3174a856d; no voters:
I20260924 07:06:17.187680 11325 raft_consensus.cc:2781] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:06:17.187758 11325 raft_consensus.cc:482] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:06:17.187804 11325 raft_consensus.cc:3037] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:17.188910 11325 raft_consensus.cc:504] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafe362852c419ba36a9014bffa7f03" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 45971 } } peers { permanent_uuid: "921e678ee49a47b5a1be6be3174a856d" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 37477 } } peers { permanent_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36493 } }
I20260924 07:06:17.189265 11325 leader_election.cc:288] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4 [CANDIDATE]: Term 1 election: Requested vote from peers 6eafe362852c419ba36a9014bffa7f03 (127.3.217.2:45971), 921e678ee49a47b5a1be6be3174a856d (127.3.217.1:37477)
I20260924 07:06:17.189563 11096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3db2dfa9c184f498ef8d9af278d61c6" candidate_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6eafe362852c419ba36a9014bffa7f03"
I20260924 07:06:17.189601 10959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3db2dfa9c184f498ef8d9af278d61c6" candidate_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "921e678ee49a47b5a1be6be3174a856d"
I20260924 07:06:17.189697 10959 raft_consensus.cc:3037] T e3db2dfa9c184f498ef8d9af278d61c6 P 921e678ee49a47b5a1be6be3174a856d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:17.189677 11096 raft_consensus.cc:3037] T e3db2dfa9c184f498ef8d9af278d61c6 P 6eafe362852c419ba36a9014bffa7f03 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:17.190583 10959 raft_consensus.cc:2454] T e3db2dfa9c184f498ef8d9af278d61c6 P 921e678ee49a47b5a1be6be3174a856d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25fe6ef8b01840ecb7cfae4d871a0ba4 in term 1.
I20260924 07:06:17.190692 11096 raft_consensus.cc:2454] T e3db2dfa9c184f498ef8d9af278d61c6 P 6eafe362852c419ba36a9014bffa7f03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25fe6ef8b01840ecb7cfae4d871a0ba4 in term 1.
I20260924 07:06:17.190824 11167 leader_election.cc:302] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25fe6ef8b01840ecb7cfae4d871a0ba4, 921e678ee49a47b5a1be6be3174a856d; no voters:
I20260924 07:06:17.190971 11325 raft_consensus.cc:2781] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:17.191177 11325 raft_consensus.cc:686] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4 [term 1 LEADER]: Becoming Leader. State: Replica: 25fe6ef8b01840ecb7cfae4d871a0ba4, State: Running, Role: LEADER
I20260924 07:06:17.191466 11325 consensus_queue.cc:237] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafe362852c419ba36a9014bffa7f03" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 45971 } } peers { permanent_uuid: "921e678ee49a47b5a1be6be3174a856d" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 37477 } } peers { permanent_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36493 } }
I20260924 07:06:17.194437 10039 catalog_manager.cc:5495] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25fe6ef8b01840ecb7cfae4d871a0ba4 (127.3.217.3). New cstate: current_term: 1 leader_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafe362852c419ba36a9014bffa7f03" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 45971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "921e678ee49a47b5a1be6be3174a856d" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 37477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36493 } health_report { overall_health: HEALTHY } } }
I20260924 07:06:17.228502 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.224680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44220 (local address 127.3.217.3:36493)
0924 07:06:17.224830 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:17.224834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:17.224847 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:06:17.224918 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:17.224920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:17.224972 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:17.225060 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:06:17.225484 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.226004 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.226962 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.227219 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.227319 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.227635 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:06:17.228087 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:06:17.228198 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:17.228300 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 07:06:17.228353 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 07:06:17.234017 10959 raft_consensus.cc:1261] T e3db2dfa9c184f498ef8d9af278d61c6 P 921e678ee49a47b5a1be6be3174a856d [term 1 FOLLOWER]: Refusing update from remote peer 25fe6ef8b01840ecb7cfae4d871a0ba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:06:17.234017 11096 raft_consensus.cc:1261] T e3db2dfa9c184f498ef8d9af278d61c6 P 6eafe362852c419ba36a9014bffa7f03 [term 1 FOLLOWER]: Refusing update from remote peer 25fe6ef8b01840ecb7cfae4d871a0ba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:06:17.234673 11325 consensus_queue.cc:1035] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "921e678ee49a47b5a1be6be3174a856d" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 37477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:06:17.234889 11326 consensus_queue.cc:1035] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6eafe362852c419ba36a9014bffa7f03" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 45971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:06:17.243439 11330 mvcc.cc:205] Tried to move back new op lower bound from 7332796732343492608 to 7332796732178423808. Current Snapshot: MvccSnapshot[applied={T|T < 7332796732343492608}]
I20260924 07:06:17.244747 11331 mvcc.cc:205] Tried to move back new op lower bound from 7332796732343492608 to 7332796732178423808. Current Snapshot: MvccSnapshot[applied={T|T < 7332796732343492608}]
I20260924 07:06:17.245016 11329 mvcc.cc:205] Tried to move back new op lower bound from 7332796732343492608 to 7332796732178423808. Current Snapshot: MvccSnapshot[applied={T|T < 7332796732343492608}]
I20260924 07:06:17.254451 10041 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:06:17.254595 10041 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:06:17.258459 10041 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c46e717f5ae4e6a9c5e6849154e5295: Sending DeleteTablet for 3 replicas of tablet e3db2dfa9c184f498ef8d9af278d61c6
I20260924 07:06:17.259747 10939 tablet_service.cc:1514] Processing DeleteTablet for tablet e3db2dfa9c184f498ef8d9af278d61c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:17 UTC) from {username='oryx', principal='oryx/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:53686
I20260924 07:06:17.260270 11340 tablet_replica.cc:330] T e3db2dfa9c184f498ef8d9af278d61c6 P 921e678ee49a47b5a1be6be3174a856d: stopping tablet replica
I20260924 07:06:17.260596 11340 raft_consensus.cc:2229] T e3db2dfa9c184f498ef8d9af278d61c6 P 921e678ee49a47b5a1be6be3174a856d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:06:17.260895 11340 raft_consensus.cc:2258] T e3db2dfa9c184f498ef8d9af278d61c6 P 921e678ee49a47b5a1be6be3174a856d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:17.261143 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 10009
I20260924 07:06:17.261181 11076 tablet_service.cc:1514] Processing DeleteTablet for tablet e3db2dfa9c184f498ef8d9af278d61c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:17 UTC) from {username='oryx', principal='oryx/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:49406
I20260924 07:06:17.261595 11212 tablet_service.cc:1514] Processing DeleteTablet for tablet e3db2dfa9c184f498ef8d9af278d61c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:17 UTC) from {username='oryx', principal='oryx/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:44214
I20260924 07:06:17.263113 11341 tablet_replica.cc:330] T e3db2dfa9c184f498ef8d9af278d61c6 P 6eafe362852c419ba36a9014bffa7f03: stopping tablet replica
I20260924 07:06:17.263830 11341 raft_consensus.cc:2229] T e3db2dfa9c184f498ef8d9af278d61c6 P 6eafe362852c419ba36a9014bffa7f03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:06:17.264456 11341 raft_consensus.cc:2258] T e3db2dfa9c184f498ef8d9af278d61c6 P 6eafe362852c419ba36a9014bffa7f03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:17.266645 11341 ts_tablet_manager.cc:1905] T e3db2dfa9c184f498ef8d9af278d61c6 P 6eafe362852c419ba36a9014bffa7f03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:06:17.269703 11341 ts_tablet_manager.cc:1918] T e3db2dfa9c184f498ef8d9af278d61c6 P 6eafe362852c419ba36a9014bffa7f03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:06:17.269874 11341 log.cc:1196] T e3db2dfa9c184f498ef8d9af278d61c6 P 6eafe362852c419ba36a9014bffa7f03: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/e3db2dfa9c184f498ef8d9af278d61c6
I20260924 07:06:17.270267 11341 ts_tablet_manager.cc:1939] T e3db2dfa9c184f498ef8d9af278d61c6 P 6eafe362852c419ba36a9014bffa7f03: Deleting consensus metadata
I20260924 07:06:17.272203 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.259914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47680 (local address 127.3.217.61:44701)
0924 07:06:17.260020 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:17.260024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:17.262861 (+ 2837us) server_negotiation.cc:407] Connection header received
0924 07:06:17.262968 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:17.262971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:17.263016 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:17.263084 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:17.264040 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.265006 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.266479 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.266829 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.266911 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.268779 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:17.268783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:17.268786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:17.268813 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:17.270389 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:17.270837 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:17.270841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:17.270843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:17.270926 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:17.271216 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:17.271219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:17.271221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:17.271449 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:17.271581 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:17.271928 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 07:06:17.272055 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:06:17.272523 11340 ts_tablet_manager.cc:1905] T e3db2dfa9c184f498ef8d9af278d61c6 P 921e678ee49a47b5a1be6be3174a856d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:06:17.275077 11340 ts_tablet_manager.cc:1918] T e3db2dfa9c184f498ef8d9af278d61c6 P 921e678ee49a47b5a1be6be3174a856d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:06:17.275159 11340 log.cc:1196] T e3db2dfa9c184f498ef8d9af278d61c6 P 921e678ee49a47b5a1be6be3174a856d: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-0/wal/wals/e3db2dfa9c184f498ef8d9af278d61c6
I20260924 07:06:17.275431 11340 ts_tablet_manager.cc:1939] T e3db2dfa9c184f498ef8d9af278d61c6 P 921e678ee49a47b5a1be6be3174a856d: Deleting consensus metadata
W20260924 07:06:17.275749 11032 connection.cc:535] server connection from 127.0.0.1:49406 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 07:06:17.275676 11342 tablet_replica.cc:330] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4: stopping tablet replica
W20260924 07:06:17.276027 10812 connection.cc:535] server connection from 127.0.0.1:52890 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 07:06:17.276242 11342 raft_consensus.cc:2229] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:06:17.276719 11342 raft_consensus.cc:2258] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:17.278767 11342 ts_tablet_manager.cc:1905] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 07:06:17.279948 10895 connection.cc:441] server connection from 127.0.0.1:53686 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53686 (request call id 1) could send its response
I20260924 07:06:17.281343 11342 ts_tablet_manager.cc:1918] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:06:17.281441 11342 log.cc:1196] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/e3db2dfa9c184f498ef8d9af278d61c6
I20260924 07:06:17.281751 11342 ts_tablet_manager.cc:1939] T e3db2dfa9c184f498ef8d9af278d61c6 P 25fe6ef8b01840ecb7cfae4d871a0ba4: Deleting consensus metadata
I20260924 07:06:17.282066 11295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.262194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52904 (local address 127.3.217.60:38295)
0924 07:06:17.262447 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:17.262452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:17.273157 (+ 10705us) server_negotiation.cc:407] Connection header received
0924 07:06:17.273322 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:17.273325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:17.273393 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:17.273441 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:17.274265 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.275066 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.276249 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.276561 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.276630 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.278210 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:17.278214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:17.278216 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:17.278241 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:17.279846 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:17.280389 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:17.280392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:17.280394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:17.280497 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:17.280796 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:17.280799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:17.280801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:17.281062 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:17.281181 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:17.281742 (+ 561us) server_negotiation.cc:299] Negotiation successful
0924 07:06:17.281905 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
W20260924 07:06:17.282788 11168 connection.cc:441] server connection from 127.0.0.1:44214 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44214 (request call id 1) could send its response
I20260924 07:06:17.288743 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.278570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47692 (local address 127.3.217.61:44701)
0924 07:06:17.278667 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:17.278671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:17.279779 (+ 1108us) server_negotiation.cc:407] Connection header received
0924 07:06:17.280009 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:17.280013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:17.280057 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:17.280109 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:17.281175 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.281901 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.283090 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.283391 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.283471 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.285474 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:17.285478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:17.285481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:17.285506 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:17.286971 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:17.287407 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:17.287411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:17.287412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:17.287490 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:17.287866 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:17.287870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:17.287871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:17.288030 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:17.288141 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:17.288430 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 07:06:17.288526 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:06:17.289790 11297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:17.278603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52912 (local address 127.3.217.60:38295)
0924 07:06:17.278693 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:17.278697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:17.281353 (+ 2656us) server_negotiation.cc:407] Connection header received
0924 07:06:17.281497 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:17.281500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:17.281540 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:17.281583 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:17.282349 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.283075 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.284121 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:17.284411 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:17.284507 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:17.285762 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:17.285766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:17.285769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:17.285791 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:17.287348 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:17.288094 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:17.288097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:17.288098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:17.288185 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:17.288517 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:17.288534 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:17.288536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:17.288814 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:17.288989 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:17.289593 (+ 604us) server_negotiation.cc:299] Negotiation successful
0924 07:06:17.289689 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:06:18.247107 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.246542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.3.217.2:50975)
0924 07:06:18.246892 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:18.246991 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":80,"threads_started":1}
I20260924 07:06:18.247346 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.246562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.3.217.1:52643)
0924 07:06:18.247085 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:18.247205 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":192,"threads_started":1}
W20260924 07:06:18.247430 11143 heartbeater.cc:644] Failed to heartbeat to 127.3.217.62:36411 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
W20260924 07:06:18.247998 11006 heartbeater.cc:644] Failed to heartbeat to 127.3.217.62:36411 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
I20260924 07:06:18.253772 11352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.253289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.3.217.3:55031)
0924 07:06:18.253592 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:18.253670 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":70,"threads_started":1}
W20260924 07:06:18.254248 11280 heartbeater.cc:644] Failed to heartbeat to 127.3.217.62:36411 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
I20260924 07:06:18.759936 11353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c46e717f5ae4e6a9c5e6849154e5295)
I20260924 07:06:18.760286 11353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: false } }
I20260924 07:06:18.761298 11353 leader_election.cc:288] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c46e717f5ae4e6a9c5e6849154e5295 (127.3.217.62:36411), 8571268ea3b24b0ca48031986cc19c2e (127.3.217.60:38295)
I20260924 07:06:18.762035 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.761544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.0.0.1:60422)
0924 07:06:18.761869 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:18.761935 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":57,"threads_started":1}
W20260924 07:06:18.762212 10629 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
I20260924 07:06:18.762889 11358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c46e717f5ae4e6a9c5e6849154e5295)
I20260924 07:06:18.763089 11358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: false } }
I20260924 07:06:18.763232 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.762961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.0.0.1:60432)
0924 07:06:18.763100 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:18.763154 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260924 07:06:18.763366 10629 leader_election.cc:334] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8c46e717f5ae4e6a9c5e6849154e5295 (127.3.217.62:36411): Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
I20260924 07:06:18.764057 11358 leader_election.cc:288] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c46e717f5ae4e6a9c5e6849154e5295 (127.3.217.62:36411), 2c9e3ac96bc54ab2a397e3e94d422074 (127.3.217.61:44701)
I20260924 07:06:18.764693 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.764165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.0.0.1:60436)
0924 07:06:18.764539 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:18.764601 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":54,"threads_started":1}
W20260924 07:06:18.764909 10812 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
I20260924 07:06:18.765726 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.765507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.0.0.1:60440)
0924 07:06:18.765614 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:18.765666 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 07:06:18.765902 10812 leader_election.cc:334] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8c46e717f5ae4e6a9c5e6849154e5295 (127.3.217.62:36411): Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
I20260924 07:06:18.766579 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.761658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.3.217.60:38295)
0924 07:06:18.761898 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:18.761903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:18.762049 (+ 146us) server_negotiation.cc:407] Connection header received
0924 07:06:18.762208 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:18.762212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:18.762263 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:18.762673 (+ 410us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:18.762800 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:18.763680 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:18.765487 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:18.766080 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:18.766171 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:18.766218 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:18.766405 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 07:06:18.766492 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":164,"thread_start_us":69,"threads_started":1}
I20260924 07:06:18.766584 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.761657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:38295 (local address 127.0.0.1:52924)
0924 07:06:18.761964 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:18.761976 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:18.762079 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:18.762560 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:18.762562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:18.762578 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:18.762673 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:18.762679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:18.763787 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:18.763790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:18.764935 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:18.764943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:18.766195 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:18.766198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:18.766285 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:18.766310 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 07:06:18.766378 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 07:06:18.766448 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":51,"threads_started":1}
I20260924 07:06:18.767059 10835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8571268ea3b24b0ca48031986cc19c2e" is_pre_election: true
I20260924 07:06:18.767246 10835 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c9e3ac96bc54ab2a397e3e94d422074 in term 1.
I20260924 07:06:18.767638 10627 leader_election.cc:302] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2c9e3ac96bc54ab2a397e3e94d422074, 8571268ea3b24b0ca48031986cc19c2e; no voters: 8c46e717f5ae4e6a9c5e6849154e5295
I20260924 07:06:18.767833 11353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 07:06:18.767890 11353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c46e717f5ae4e6a9c5e6849154e5295)
I20260924 07:06:18.767926 11353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:06:18.768524 11361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.764287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47694 (local address 127.3.217.61:44701)
0924 07:06:18.764543 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:18.764550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:18.765027 (+ 477us) server_negotiation.cc:407] Connection header received
0924 07:06:18.765063 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:18.765066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:18.765104 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:18.765173 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:18.765558 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:18.766184 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:18.767479 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:18.768037 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:18.768132 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:18.768184 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:18.768360 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 07:06:18.768427 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":104,"threads_started":1}
I20260924 07:06:18.768524 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.764223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:44701 (local address 127.0.0.1:47694)
0924 07:06:18.764820 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:18.764832 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:18.764917 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:18.765199 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:18.765202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:18.765310 (+ 108us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:18.765394 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:18.765439 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:18.766299 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:18.766302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:18.767377 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:18.767386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:18.768158 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:18.768162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:18.768248 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:18.768268 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 07:06:18.768332 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 07:06:18.768404 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":40,"threads_started":1}
I20260924 07:06:18.768908 10653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8571268ea3b24b0ca48031986cc19c2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" is_pre_election: true
I20260924 07:06:18.769176 11353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: false } }
I20260924 07:06:18.769450 10653 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8571268ea3b24b0ca48031986cc19c2e in current term 2: Already voted for candidate 2c9e3ac96bc54ab2a397e3e94d422074 in this term.
I20260924 07:06:18.769508 11353 leader_election.cc:288] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [CANDIDATE]: Term 2 election: Requested vote from peers 8c46e717f5ae4e6a9c5e6849154e5295 (127.3.217.62:36411), 8571268ea3b24b0ca48031986cc19c2e (127.3.217.60:38295)
I20260924 07:06:18.769842 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.769632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.0.0.1:60448)
0924 07:06:18.769739 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:18.769786 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 07:06:18.769832 10811 leader_election.cc:302] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8571268ea3b24b0ca48031986cc19c2e; no voters: 2c9e3ac96bc54ab2a397e3e94d422074, 8c46e717f5ae4e6a9c5e6849154e5295
I20260924 07:06:18.769939 10835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8571268ea3b24b0ca48031986cc19c2e"
I20260924 07:06:18.770058 10835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:06:18.770385 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.770207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.0.0.1:60460)
0924 07:06:18.770294 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:18.770338 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 07:06:18.770468 10629 leader_election.cc:334] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8c46e717f5ae4e6a9c5e6849154e5295 (127.3.217.62:36411): Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
I20260924 07:06:18.770908 10835 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c9e3ac96bc54ab2a397e3e94d422074 in term 2.
I20260924 07:06:18.771013 11358 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:06:18.771449 10627 leader_election.cc:302] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2c9e3ac96bc54ab2a397e3e94d422074, 8571268ea3b24b0ca48031986cc19c2e; no voters: 8c46e717f5ae4e6a9c5e6849154e5295
I20260924 07:06:18.771641 11353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 07:06:18.771834 11353 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [term 2 LEADER]: Becoming Leader. State: Replica: 2c9e3ac96bc54ab2a397e3e94d422074, State: Running, Role: LEADER
I20260924 07:06:18.772053 11353 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: false } }
I20260924 07:06:18.773203 11363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c9e3ac96bc54ab2a397e3e94d422074. Latest consensus state: current_term: 2 leader_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: false } } }
I20260924 07:06:18.773336 11363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:18.773773 11365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:06:18.776379 11365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:06:18.776772 11365 catalog_manager.cc:1259] Loaded cluster ID: 5730979573034234b8f25895f606f0bd
I20260924 07:06:18.776834 11365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:06:18.777390 11365 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:06:18.777727 11365 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074: Loaded TSK: 0
I20260924 07:06:18.778386 11365 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:06:18.805820 10643 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:06:18.808921 10643 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:06:18.812633 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.812395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.0.0.1:60472)
0924 07:06:18.812511 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:18.812564 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 07:06:18.812747 10835 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [term 2 FOLLOWER]: Refusing update from remote peer 2c9e3ac96bc54ab2a397e3e94d422074: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 07:06:18.813413 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.813041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.0.0.1:60484)
0924 07:06:18.813215 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:18.813302 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260924 07:06:18.813344 11363 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260924 07:06:18.813716 10629 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 -> Peer 8c46e717f5ae4e6a9c5e6849154e5295 (127.3.217.62:36411): Couldn't send request to peer 8c46e717f5ae4e6a9c5e6849154e5295. Status: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 07:06:18.815107 11358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c9e3ac96bc54ab2a397e3e94d422074. Latest consensus state: current_term: 2 leader_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: false } } }
I20260924 07:06:18.815239 11358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:06:18.816392 11363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: false } } }
I20260924 07:06:18.816715 11363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:18.817219 11358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c46e717f5ae4e6a9c5e6849154e5295" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36411 } } peers { permanent_uuid: "2c9e3ac96bc54ab2a397e3e94d422074" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 44701 } attrs { promote: false } } peers { permanent_uuid: "8571268ea3b24b0ca48031986cc19c2e" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 38295 } attrs { promote: false } } }
I20260924 07:06:18.817374 11358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8571268ea3b24b0ca48031986cc19c2e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:06:18.827479 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.823000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:37477 (local address 127.0.0.1:53694)
0924 07:06:18.823187 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:18.823204 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:18.823314 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:18.824013 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:18.824015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:18.824033 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:18.824113 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:18.824118 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:18.824964 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:18.824968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:18.826208 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:18.826219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:18.827076 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:18.827080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:18.827173 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:18.827196 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:06:18.827262 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 07:06:18.827343 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":58}
I20260924 07:06:18.827613 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.822758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:45971 (local address 127.0.0.1:49414)
0924 07:06:18.822917 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:18.822931 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:18.823012 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:18.823672 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:18.823675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:18.823692 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:18.823822 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:18.823829 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:18.824897 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:18.824901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:18.825879 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:18.825886 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:18.826664 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:18.826667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:18.827234 (+ 567us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:18.827260 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 07:06:18.827341 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 07:06:18.827414 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":357}
I20260924 07:06:18.828150 11372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.822915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49414 (local address 127.3.217.2:45971)
0924 07:06:18.823330 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:18.823335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:18.823351 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:06:18.823410 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:18.823413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:18.823463 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:18.823929 (+ 466us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:18.824036 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:18.824781 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:18.826020 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:18.826552 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:18.826658 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:18.826728 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:18.827933 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0924 07:06:18.828023 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":113,"threads_started":1}
I20260924 07:06:18.828202 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.823492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53694 (local address 127.3.217.1:37477)
0924 07:06:18.823788 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:18.823791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:18.823803 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:06:18.823845 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:18.823847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:18.823897 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:18.823984 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:18.824232 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:18.824844 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:18.826480 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:18.826961 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:18.827053 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:18.827110 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:18.827933 (+ 823us) server_negotiation.cc:299] Negotiation successful
0924 07:06:18.828058 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
I20260924 07:06:18.828735 10939 tablet_service.cc:1467] Processing CreateTablet for tablet d2a3841d633e4eb1b1d74ae63fcc86ee (DEFAULT_TABLE table=test-table [id=b97a1659dcab4093be04188a2b26def6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:18.828821 11076 tablet_service.cc:1467] Processing CreateTablet for tablet d2a3841d633e4eb1b1d74ae63fcc86ee (DEFAULT_TABLE table=test-table [id=b97a1659dcab4093be04188a2b26def6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:18.829026 10939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a3841d633e4eb1b1d74ae63fcc86ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:18.829151 11076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a3841d633e4eb1b1d74ae63fcc86ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:18.830180 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.823238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:36493 (local address 127.0.0.1:44230)
0924 07:06:18.823467 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:18.823480 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:18.823644 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:18.825285 (+ 1641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:18.825288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:18.825308 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:18.825426 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:18.825431 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:18.826546 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:18.826549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:18.828942 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:18.828952 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:18.829792 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:18.829796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:18.829891 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:18.829913 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:06:18.829992 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 07:06:18.830056 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":5,"thread_start_us":92,"threads_started":1}
I20260924 07:06:18.830245 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.823495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44230 (local address 127.3.217.3:36493)
0924 07:06:18.825018 (+ 1523us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:18.825023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:18.825037 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:06:18.825092 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:18.825095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:18.825147 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:18.825252 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:18.825551 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:18.826411 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:18.829095 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:18.829662 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:18.829765 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:18.829827 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:18.830035 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 07:06:18.830120 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1383,"thread_start_us":82,"threads_started":1}
I20260924 07:06:18.831370 11376 tablet_bootstrap.cc:492] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d: Bootstrap starting.
I20260924 07:06:18.831830 11212 tablet_service.cc:1467] Processing CreateTablet for tablet d2a3841d633e4eb1b1d74ae63fcc86ee (DEFAULT_TABLE table=test-table [id=b97a1659dcab4093be04188a2b26def6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:18.832077 11212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a3841d633e4eb1b1d74ae63fcc86ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:18.832307 11376 tablet_bootstrap.cc:654] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:18.833698 11376 tablet_bootstrap.cc:492] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d: No bootstrap required, opened a new log
I20260924 07:06:18.833864 11376 ts_tablet_manager.cc:1397] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 07:06:18.834575 11376 raft_consensus.cc:348] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafe362852c419ba36a9014bffa7f03" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 45971 } } peers { permanent_uuid: "921e678ee49a47b5a1be6be3174a856d" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 37477 } } peers { permanent_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36493 } }
I20260924 07:06:18.834764 11376 raft_consensus.cc:374] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:18.834800 11376 raft_consensus.cc:729] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 921e678ee49a47b5a1be6be3174a856d, State: Initialized, Role: FOLLOWER
I20260924 07:06:18.834960 11376 consensus_queue.cc:260] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafe362852c419ba36a9014bffa7f03" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 45971 } } peers { permanent_uuid: "921e678ee49a47b5a1be6be3174a856d" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 37477 } } peers { permanent_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36493 } }
I20260924 07:06:18.835125 11379 tablet_bootstrap.cc:492] T d2a3841d633e4eb1b1d74ae63fcc86ee P 25fe6ef8b01840ecb7cfae4d871a0ba4: Bootstrap starting.
I20260924 07:06:18.835472 11376 ts_tablet_manager.cc:1428] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:06:18.836078 11379 tablet_bootstrap.cc:654] T d2a3841d633e4eb1b1d74ae63fcc86ee P 25fe6ef8b01840ecb7cfae4d871a0ba4: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:18.836831 11381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.835930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.3.217.1:59589)
0924 07:06:18.836657 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:18.836713 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":111,"threads_started":1}
I20260924 07:06:18.837152 11379 tablet_bootstrap.cc:492] T d2a3841d633e4eb1b1d74ae63fcc86ee P 25fe6ef8b01840ecb7cfae4d871a0ba4: No bootstrap required, opened a new log
I20260924 07:06:18.837231 11379 ts_tablet_manager.cc:1397] T d2a3841d633e4eb1b1d74ae63fcc86ee P 25fe6ef8b01840ecb7cfae4d871a0ba4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:06:18.837576 11379 raft_consensus.cc:348] T d2a3841d633e4eb1b1d74ae63fcc86ee P 25fe6ef8b01840ecb7cfae4d871a0ba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafe362852c419ba36a9014bffa7f03" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 45971 } } peers { permanent_uuid: "921e678ee49a47b5a1be6be3174a856d" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 37477 } } peers { permanent_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36493 } }
I20260924 07:06:18.837677 11379 raft_consensus.cc:374] T d2a3841d633e4eb1b1d74ae63fcc86ee P 25fe6ef8b01840ecb7cfae4d871a0ba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:18.837713 11379 raft_consensus.cc:729] T d2a3841d633e4eb1b1d74ae63fcc86ee P 25fe6ef8b01840ecb7cfae4d871a0ba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25fe6ef8b01840ecb7cfae4d871a0ba4, State: Initialized, Role: FOLLOWER
I20260924 07:06:18.837829 11381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.837188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.3.217.1:38271)
0924 07:06:18.837431 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:18.837636 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
I20260924 07:06:18.837850 11379 consensus_queue.cc:260] T d2a3841d633e4eb1b1d74ae63fcc86ee P 25fe6ef8b01840ecb7cfae4d871a0ba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafe362852c419ba36a9014bffa7f03" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 45971 } } peers { permanent_uuid: "921e678ee49a47b5a1be6be3174a856d" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 37477 } } peers { permanent_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36493 } }
I20260924 07:06:18.838279 11379 ts_tablet_manager.cc:1428] T d2a3841d633e4eb1b1d74ae63fcc86ee P 25fe6ef8b01840ecb7cfae4d871a0ba4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:06:18.838773 11007 heartbeater.cc:503] Master 127.3.217.61:44701 was elected leader, sending a full tablet report...
I20260924 07:06:18.839289 11385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.838782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.3.217.3:47181)
0924 07:06:18.839148 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:18.839205 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":93,"threads_started":1}
I20260924 07:06:18.840323 11377 tablet_bootstrap.cc:492] T d2a3841d633e4eb1b1d74ae63fcc86ee P 6eafe362852c419ba36a9014bffa7f03: Bootstrap starting.
I20260924 07:06:18.841156 11377 tablet_bootstrap.cc:654] T d2a3841d633e4eb1b1d74ae63fcc86ee P 6eafe362852c419ba36a9014bffa7f03: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:18.842149 11377 tablet_bootstrap.cc:492] T d2a3841d633e4eb1b1d74ae63fcc86ee P 6eafe362852c419ba36a9014bffa7f03: No bootstrap required, opened a new log
I20260924 07:06:18.842221 11377 ts_tablet_manager.cc:1397] T d2a3841d633e4eb1b1d74ae63fcc86ee P 6eafe362852c419ba36a9014bffa7f03: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:06:18.842527 11377 raft_consensus.cc:348] T d2a3841d633e4eb1b1d74ae63fcc86ee P 6eafe362852c419ba36a9014bffa7f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafe362852c419ba36a9014bffa7f03" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 45971 } } peers { permanent_uuid: "921e678ee49a47b5a1be6be3174a856d" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 37477 } } peers { permanent_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36493 } }
I20260924 07:06:18.842614 11377 raft_consensus.cc:374] T d2a3841d633e4eb1b1d74ae63fcc86ee P 6eafe362852c419ba36a9014bffa7f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:18.842634 11377 raft_consensus.cc:729] T d2a3841d633e4eb1b1d74ae63fcc86ee P 6eafe362852c419ba36a9014bffa7f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eafe362852c419ba36a9014bffa7f03, State: Initialized, Role: FOLLOWER
I20260924 07:06:18.842730 11377 consensus_queue.cc:260] T d2a3841d633e4eb1b1d74ae63fcc86ee P 6eafe362852c419ba36a9014bffa7f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafe362852c419ba36a9014bffa7f03" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 45971 } } peers { permanent_uuid: "921e678ee49a47b5a1be6be3174a856d" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 37477 } } peers { permanent_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36493 } }
I20260924 07:06:18.843340 11377 ts_tablet_manager.cc:1428] T d2a3841d633e4eb1b1d74ae63fcc86ee P 6eafe362852c419ba36a9014bffa7f03: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:06:18.843737 11388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.843331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.3.217.2:48365)
0924 07:06:18.843594 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:18.843645 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":83,"threads_started":1}
I20260924 07:06:18.844413 11144 heartbeater.cc:503] Master 127.3.217.61:44701 was elected leader, sending a full tablet report...
I20260924 07:06:18.844436 11389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:18.844105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.3.217.2:46591)
0924 07:06:18.844299 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:18.844348 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
W20260924 07:06:19.033252 11282 tablet.cc:2369] T d2a3841d633e4eb1b1d74ae63fcc86ee P 25fe6ef8b01840ecb7cfae4d871a0ba4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:06:19.059887 11145 tablet.cc:2369] T d2a3841d633e4eb1b1d74ae63fcc86ee P 6eafe362852c419ba36a9014bffa7f03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:06:19.084241 11380 raft_consensus.cc:482] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:06:19.084389 11380 raft_consensus.cc:504] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafe362852c419ba36a9014bffa7f03" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 45971 } } peers { permanent_uuid: "921e678ee49a47b5a1be6be3174a856d" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 37477 } } peers { permanent_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36493 } }
I20260924 07:06:19.085832 11380 leader_election.cc:288] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6eafe362852c419ba36a9014bffa7f03 (127.3.217.2:45971), 25fe6ef8b01840ecb7cfae4d871a0ba4 (127.3.217.3:36493)
I20260924 07:06:19.090896 11381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:19.086372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:36493 (local address 127.3.217.1:59545)
0924 07:06:19.086480 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:19.086491 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:19.086667 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:19.086984 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:19.086988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:19.087016 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:19.087166 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:19.087174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:19.088291 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:19.088296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:19.089271 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:19.089281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:19.090450 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:19.090452 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:19.090573 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:19.090595 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:06:19.090675 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 07:06:19.090768 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":7}
I20260924 07:06:19.090896 11372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:19.086209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:53671 (local address 127.3.217.2:45971)
0924 07:06:19.086337 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:19.086341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:19.086356 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:06:19.086538 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:19.086542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:19.086592 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:19.086927 (+ 335us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:19.087688 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:19.088243 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:19.089746 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:19.090164 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:19.090252 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:19.090309 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:19.090686 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 07:06:19.090768 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:06:19.091104 11383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:19.086120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:45971 (local address 127.3.217.1:53671)
0924 07:06:19.086247 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:19.086264 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:19.086403 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:19.086912 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:19.086914 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:19.086931 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:19.087213 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:19.087230 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:19.088487 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:19.088490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:19.089320 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:19.089328 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:19.090349 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:19.090354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:19.090529 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:19.090558 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 07:06:19.090632 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 07:06:19.090914 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":80}
I20260924 07:06:19.091089 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:19.086484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:59545 (local address 127.3.217.3:36493)
0924 07:06:19.086683 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:19.086687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:19.086702 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:06:19.086761 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:19.086763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:19.086809 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:19.086936 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:19.087394 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:19.088132 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:19.089627 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:19.090272 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:19.090410 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:19.090492 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:19.090843 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 07:06:19.090947 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:06:19.091786 11233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a3841d633e4eb1b1d74ae63fcc86ee" candidate_uuid: "921e678ee49a47b5a1be6be3174a856d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" is_pre_election: true
I20260924 07:06:19.092027 11233 raft_consensus.cc:2454] T d2a3841d633e4eb1b1d74ae63fcc86ee P 25fe6ef8b01840ecb7cfae4d871a0ba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 921e678ee49a47b5a1be6be3174a856d in term 0.
I20260924 07:06:19.092248 11096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a3841d633e4eb1b1d74ae63fcc86ee" candidate_uuid: "921e678ee49a47b5a1be6be3174a856d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6eafe362852c419ba36a9014bffa7f03" is_pre_election: true
I20260924 07:06:19.092466 11096 raft_consensus.cc:2454] T d2a3841d633e4eb1b1d74ae63fcc86ee P 6eafe362852c419ba36a9014bffa7f03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 921e678ee49a47b5a1be6be3174a856d in term 0.
I20260924 07:06:19.092514 10892 leader_election.cc:302] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25fe6ef8b01840ecb7cfae4d871a0ba4, 921e678ee49a47b5a1be6be3174a856d; no voters:
I20260924 07:06:19.092782 11380 raft_consensus.cc:2781] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:06:19.092844 11380 raft_consensus.cc:482] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:06:19.092880 11380 raft_consensus.cc:3037] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:19.093880 11380 raft_consensus.cc:504] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafe362852c419ba36a9014bffa7f03" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 45971 } } peers { permanent_uuid: "921e678ee49a47b5a1be6be3174a856d" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 37477 } } peers { permanent_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36493 } }
I20260924 07:06:19.094187 11380 leader_election.cc:288] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d [CANDIDATE]: Term 1 election: Requested vote from peers 6eafe362852c419ba36a9014bffa7f03 (127.3.217.2:45971), 25fe6ef8b01840ecb7cfae4d871a0ba4 (127.3.217.3:36493)
I20260924 07:06:19.094817 11096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a3841d633e4eb1b1d74ae63fcc86ee" candidate_uuid: "921e678ee49a47b5a1be6be3174a856d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6eafe362852c419ba36a9014bffa7f03"
I20260924 07:06:19.094851 11233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a3841d633e4eb1b1d74ae63fcc86ee" candidate_uuid: "921e678ee49a47b5a1be6be3174a856d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4"
I20260924 07:06:19.094956 11096 raft_consensus.cc:3037] T d2a3841d633e4eb1b1d74ae63fcc86ee P 6eafe362852c419ba36a9014bffa7f03 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:19.094956 11233 raft_consensus.cc:3037] T d2a3841d633e4eb1b1d74ae63fcc86ee P 25fe6ef8b01840ecb7cfae4d871a0ba4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:19.096047 11096 raft_consensus.cc:2454] T d2a3841d633e4eb1b1d74ae63fcc86ee P 6eafe362852c419ba36a9014bffa7f03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 921e678ee49a47b5a1be6be3174a856d in term 1.
I20260924 07:06:19.096097 11233 raft_consensus.cc:2454] T d2a3841d633e4eb1b1d74ae63fcc86ee P 25fe6ef8b01840ecb7cfae4d871a0ba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 921e678ee49a47b5a1be6be3174a856d in term 1.
I20260924 07:06:19.096324 10892 leader_election.cc:302] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25fe6ef8b01840ecb7cfae4d871a0ba4, 921e678ee49a47b5a1be6be3174a856d; no voters:
I20260924 07:06:19.096498 11380 raft_consensus.cc:2781] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:19.096706 11380 raft_consensus.cc:686] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d [term 1 LEADER]: Becoming Leader. State: Replica: 921e678ee49a47b5a1be6be3174a856d, State: Running, Role: LEADER
I20260924 07:06:19.096935 11380 consensus_queue.cc:237] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafe362852c419ba36a9014bffa7f03" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 45971 } } peers { permanent_uuid: "921e678ee49a47b5a1be6be3174a856d" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 37477 } } peers { permanent_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36493 } }
I20260924 07:06:19.097944 11383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:19.097545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.3.217.1:36119)
0924 07:06:19.097762 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:19.097837 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260924 07:06:19.099617 10642 catalog_manager.cc:5495] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d reported cstate change: term changed from 0 to 1, leader changed from <none> to 921e678ee49a47b5a1be6be3174a856d (127.3.217.1). New cstate: current_term: 1 leader_uuid: "921e678ee49a47b5a1be6be3174a856d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafe362852c419ba36a9014bffa7f03" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 45971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "921e678ee49a47b5a1be6be3174a856d" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 37477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36493 } health_report { overall_health: UNKNOWN } } }
I20260924 07:06:19.133416 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:19.128736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53702 (local address 127.3.217.1:37477)
0924 07:06:19.128926 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:19.128931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:19.128954 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:06:19.129110 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:19.129112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:19.129173 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:19.129291 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:06:19.129744 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:19.130333 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:19.131303 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:19.131582 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:19.131733 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:19.132079 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:06:19.132732 (+ 653us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:06:19.132844 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:19.133150 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 07:06:19.133238 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 07:06:19.136693 11233 raft_consensus.cc:1261] T d2a3841d633e4eb1b1d74ae63fcc86ee P 25fe6ef8b01840ecb7cfae4d871a0ba4 [term 1 FOLLOWER]: Refusing update from remote peer 921e678ee49a47b5a1be6be3174a856d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:06:19.136718 11096 raft_consensus.cc:1261] T d2a3841d633e4eb1b1d74ae63fcc86ee P 6eafe362852c419ba36a9014bffa7f03 [term 1 FOLLOWER]: Refusing update from remote peer 921e678ee49a47b5a1be6be3174a856d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:06:19.137486 11390 consensus_queue.cc:1035] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6eafe362852c419ba36a9014bffa7f03" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 45971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:06:19.137763 11380 consensus_queue.cc:1035] T d2a3841d633e4eb1b1d74ae63fcc86ee P 921e678ee49a47b5a1be6be3174a856d [LEADER]: Connected to new peer: Peer: permanent_uuid: "25fe6ef8b01840ecb7cfae4d871a0ba4" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 36493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:06:19.139220 11281 heartbeater.cc:503] Master 127.3.217.61:44701 was elected leader, sending a full tablet report...
I20260924 07:06:19.140137 11389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:19.139780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.3.217.2:56273)
0924 07:06:19.139990 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:19.140052 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260924 07:06:19.140985 11385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:19.140443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.3.217.3:41113)
0924 07:06:19.140852 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:19.140911 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338}
I20260924 07:06:19.143653 11383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:19.143341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.3.217.1:43667)
0924 07:06:19.143486 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:19.143580 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 07:06:19.145095 11389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:19.144232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.3.217.2:34759)
0924 07:06:19.144983 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:19.145035 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":683}
I20260924 07:06:19.146427 11383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:19.146219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.3.217.1:50163)
0924 07:06:19.146339 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:19.146377 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 07:06:19.148447 11385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:19.147539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:36411 (local address 127.3.217.3:37553)
0924 07:06:19.148307 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:19.148369 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527}
I20260924 07:06:19.152010 10643 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:06:19.152166 10643 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:06:19.156765 10643 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c9e3ac96bc54ab2a397e3e94d422074: Sending DeleteTablet for 3 replicas of tablet d2a3841d633e4eb1b1d74ae63fcc86ee
I20260924 07:06:19.157948 10939 tablet_service.cc:1514] Processing DeleteTablet for tablet d2a3841d633e4eb1b1d74ae63fcc86ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:19 UTC) from {username='oryx', principal='oryx/127.3.217.61@KRBTEST.COM'} at 127.0.0.1:53694
I20260924 07:06:19.158174 11212 tablet_service.cc:1514] Processing DeleteTablet for tablet d2a3841d633e4eb1b1d74ae63fcc86ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:19 UTC) from {username='oryx', principal='oryx/127.3.217.61@KRBTEST.COM'} at 127.0.0.1:44230
I20260924 07:06:19.158376 11076 tablet_service.cc:1514] Processing DeleteTablet for tablet d2a3841d633e4eb1b1d74ae63fcc86ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:19 UTC) from {username='oryx', principal='oryx/127.3.217.61@KRBTEST.COM'} at 127.0.0.1:49414
I20260924 07:06:19.158933 11404 tablet_replica.cc:330] T d2a3841d633e4eb1b1d74ae63fcc86ee P 25fe6ef8b01840ecb7cfae4d871a0ba4: stopping tablet replica
I20260924 07:06:19.159065 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 10788
I20260924 07:06:19.160095 11405 tablet_replica.cc:330] T d2a3841d633e4eb1b1d74ae63fcc86ee P 6eafe362852c419ba36a9014bffa7f03: stopping tablet replica
I20260924 07:06:19.160221 11405 raft_consensus.cc:2229] T d2a3841d633e4eb1b1d74ae63fcc86ee P 6eafe362852c419ba36a9014bffa7f03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:06:19.160359 11405 raft_consensus.cc:2258] T d2a3841d633e4eb1b1d74ae63fcc86ee P 6eafe362852c419ba36a9014bffa7f03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:19.160478 11404 raft_consensus.cc:2229] T d2a3841d633e4eb1b1d74ae63fcc86ee P 25fe6ef8b01840ecb7cfae4d871a0ba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:06:19.160625 11404 raft_consensus.cc:2258] T d2a3841d633e4eb1b1d74ae63fcc86ee P 25fe6ef8b01840ecb7cfae4d871a0ba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:19.161273 11404 ts_tablet_manager.cc:1905] T d2a3841d633e4eb1b1d74ae63fcc86ee P 25fe6ef8b01840ecb7cfae4d871a0ba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:06:19.161626 11405 ts_tablet_manager.cc:1905] T d2a3841d633e4eb1b1d74ae63fcc86ee P 6eafe362852c419ba36a9014bffa7f03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:06:19.163887 11404 ts_tablet_manager.cc:1918] T d2a3841d633e4eb1b1d74ae63fcc86ee P 25fe6ef8b01840ecb7cfae4d871a0ba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:06:19.163967 11404 log.cc:1196] T d2a3841d633e4eb1b1d74ae63fcc86ee P 25fe6ef8b01840ecb7cfae4d871a0ba4: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/d2a3841d633e4eb1b1d74ae63fcc86ee
I20260924 07:06:19.163931 11405 ts_tablet_manager.cc:1918] T d2a3841d633e4eb1b1d74ae63fcc86ee P 6eafe362852c419ba36a9014bffa7f03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:06:19.164173 11405 log.cc:1196] T d2a3841d633e4eb1b1d74ae63fcc86ee P 6eafe362852c419ba36a9014bffa7f03: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/d2a3841d633e4eb1b1d74ae63fcc86ee
I20260924 07:06:19.164229 11404 ts_tablet_manager.cc:1939] T d2a3841d633e4eb1b1d74ae63fcc86ee P 25fe6ef8b01840ecb7cfae4d871a0ba4: Deleting consensus metadata
I20260924 07:06:19.164516 11405 ts_tablet_manager.cc:1939] T d2a3841d633e4eb1b1d74ae63fcc86ee P 6eafe362852c419ba36a9014bffa7f03: Deleting consensus metadata
I20260924 07:06:19.165040 10627 catalog_manager.cc:4841] TS 25fe6ef8b01840ecb7cfae4d871a0ba4 (127.3.217.3:36493): tablet d2a3841d633e4eb1b1d74ae63fcc86ee (table test-table [id=b97a1659dcab4093be04188a2b26def6]) successfully deleted
I20260924 07:06:19.165238 10627 catalog_manager.cc:4841] TS 6eafe362852c419ba36a9014bffa7f03 (127.3.217.2:45971): tablet d2a3841d633e4eb1b1d74ae63fcc86ee (table test-table [id=b97a1659dcab4093be04188a2b26def6]) successfully deleted
I20260924 07:06:19.165750 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:19.165517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:37477 (local address 127.0.0.1:53710)
0924 07:06:19.165641 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:19.165689 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.1:37477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 07:06:19.165753 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 11013
W20260924 07:06:19.166817 10629 catalog_manager.cc:4584] TS 921e678ee49a47b5a1be6be3174a856d (127.3.217.1:37477): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2a3841d633e4eb1b1d74ae63fcc86ee: Network error: Client connection negotiation failed: client connection to 127.3.217.1:37477: connect: Connection refused (error 111)
I20260924 07:06:19.171123 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 11149
I20260924 07:06:19.177421 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 10505
I20260924 07:06:19.182231 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 10608
2026-09-24T07:06:19Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5503 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:06:19 dist-test-slave-2p7j krb5kdc[11408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:06:19 dist-test-slave-2p7j krb5kdc[11408](info): set up 2 sockets
Sep 24 07:06:19 dist-test-slave-2p7j krb5kdc[11408](info): commencing operation
krb5kdc: starting...
W20260924 07:06:20.180301 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:06:20 dist-test-slave-2p7j krb5kdc[11408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:06:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:06:20Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.62.keytab.
Entry for principal oryx/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.62.keytab.
I20260924 07:06:20.312017 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:43511
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.62.keytab
--principal=oryx/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:32867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:43511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:20.420987 11421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:20.421231 11421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:20.421284 11421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:20.425199 11421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:06:20.425252 11421 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 07:06:20.425273 11421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:20.425297 11421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:20.425318 11421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:06:20.425338 11421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:06:20.430395 11421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:32867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:43511
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.62.keytab
--principal=oryx/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:43511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:20.431324 11421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:20.432536 11421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:20.441143 11426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:20.441115 11429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:20.441198 11421 server_base.cc:1034] running on GCE node
W20260924 07:06:20.441125 11427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:20.441910 11421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:20.442890 11421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:20.444084 11421 hybrid_clock.cc:648] HybridClock initialized: now 1790233580444070 us; error 52 us; skew 500 ppm
Sep 24 07:06:20 dist-test-slave-2p7j krb5kdc[11408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233580, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:20.447238 11421 init.cc:377] Logged in from keytab as oryx/127.3.217.62@KRBTEST.COM (short username oryx)
I20260924 07:06:20.448417 11421 webserver.cc:460] Webserver started at http://127.3.217.62:44015/ using document root <none> and password file <none>
I20260924 07:06:20.449091 11421 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:20.449167 11421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:20.449479 11421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:20.452281 11421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "5a8fc551150745cca69f43e7273ce040"
format_stamp: "Formatted at 2026-09-24 07:06:20 on dist-test-slave-2p7j"
server_key: "062fa2f993e09aea358ff981b11103b6"
server_key_iv: "e349f3857eb0a5a1babccb29ff8e070c"
server_key_version: "encryptionkey@0"
I20260924 07:06:20.452877 11421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "5a8fc551150745cca69f43e7273ce040"
format_stamp: "Formatted at 2026-09-24 07:06:20 on dist-test-slave-2p7j"
server_key: "062fa2f993e09aea358ff981b11103b6"
server_key_iv: "e349f3857eb0a5a1babccb29ff8e070c"
server_key_version: "encryptionkey@0"
I20260924 07:06:20.456403 11421 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 07:06:20.459120 11436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:20.460485 11421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:06:20.460613 11421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "5a8fc551150745cca69f43e7273ce040"
format_stamp: "Formatted at 2026-09-24 07:06:20 on dist-test-slave-2p7j"
server_key: "062fa2f993e09aea358ff981b11103b6"
server_key_iv: "e349f3857eb0a5a1babccb29ff8e070c"
server_key_version: "encryptionkey@0"
I20260924 07:06:20.460713 11421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:20.473773 11421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:20.476955 11421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:20.477125 11421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:20.486111 11421 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:43511
I20260924 07:06:20.486117 11499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:43511 every 8 connection(s)
I20260924 07:06:20.487088 11421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:06:20.489413 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 11421
I20260924 07:06:20.489534 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:06:20.489785 3940 external_mini_cluster.cc:1428] Setting key 2c0588d3b9cab0c01fa5d3ab9b3b299c
I20260924 07:06:20.491480 11500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:20.498556 11500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040: Bootstrap starting.
Sep 24 07:06:20 dist-test-slave-2p7j krb5kdc[11408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM
I20260924 07:06:20.500878 11500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:20.502038 11500 log.cc:824] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:20.504189 11500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040: No bootstrap required, opened a new log
I20260924 07:06:20.505399 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:20.491279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54394 (local address 127.3.217.62:43511)
0924 07:06:20.491828 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:20.491851 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:20.491908 (+ 57us) server_negotiation.cc:407] Connection header received
0924 07:06:20.493092 (+ 1184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:20.493116 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:20.493556 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:20.493920 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:20.494957 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:20.496419 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:20.497300 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:20.497971 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:20.498121 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:20.500826 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:20.500831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:20.500843 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:20.500869 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:20.502562 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:20.503073 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:20.503077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:20.503082 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:20.503185 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:20.503596 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:20.503604 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:20.503606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:20.503977 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:20.504131 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:20.504434 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 07:06:20.504704 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":167,"threads_started":1}
I20260924 07:06:20.507073 11500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8fc551150745cca69f43e7273ce040" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 43511 } }
I20260924 07:06:20.507285 11500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:20.507361 11500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a8fc551150745cca69f43e7273ce040, State: Initialized, Role: FOLLOWER
I20260924 07:06:20.507799 11500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8fc551150745cca69f43e7273ce040" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 43511 } }
I20260924 07:06:20.507927 11500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:06:20.508000 11500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:06:20.508090 11500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:20.508889 11500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8fc551150745cca69f43e7273ce040" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 43511 } }
I20260924 07:06:20.509217 11500 leader_election.cc:302] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a8fc551150745cca69f43e7273ce040; no voters:
I20260924 07:06:20.509503 11500 leader_election.cc:288] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:06:20.509794 11505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:20.510185 11505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040 [term 1 LEADER]: Becoming Leader. State: Replica: 5a8fc551150745cca69f43e7273ce040, State: Running, Role: LEADER
I20260924 07:06:20.510561 11500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:06:20.510690 11505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8fc551150745cca69f43e7273ce040" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 43511 } }
I20260924 07:06:20.512275 11507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a8fc551150745cca69f43e7273ce040. Latest consensus state: current_term: 1 leader_uuid: "5a8fc551150745cca69f43e7273ce040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8fc551150745cca69f43e7273ce040" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 43511 } } }
I20260924 07:06:20.512393 11507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:20.512957 11506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a8fc551150745cca69f43e7273ce040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8fc551150745cca69f43e7273ce040" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 43511 } } }
I20260924 07:06:20.513067 11506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040 [sys.catalog]: This master's current role is: LEADER
W20260924 07:06:20.516047 11519 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:06:20.516126 11519 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:06:20.516207 11520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:06:20.517056 11520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:06:20.522003 11520 catalog_manager.cc:1347] Generated new cluster ID: 753b99100ba249beb3a7fb2899c6dd45
I20260924 07:06:20.522069 11520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:06:20.533120 11520 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:06:20.534068 11520 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:06:20.551873 11520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040: Generated new TSK 0
I20260924 07:06:20.553149 11520 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.1.keytab.
Entry for principal oryx/127.3.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.1.keytab.
I20260924 07:06:20.593655 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:43511
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.1.keytab
--principal=oryx/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:32867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:20.712435 11526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:20.712957 11526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:20.713101 11526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:20.719762 11526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:20.719825 11526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:20.719869 11526 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:06:20.719910 11526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:06:20.725834 11526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:32867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.1.keytab
--principal=oryx/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.217.62:43511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:20.726809 11526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:20.727773 11526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:20.738181 11532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:20.738368 11526 server_base.cc:1034] running on GCE node
W20260924 07:06:20.738282 11531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:20.738186 11534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:20.739043 11526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:20.739724 11526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:20.740931 11526 hybrid_clock.cc:648] HybridClock initialized: now 1790233580740873 us; error 72 us; skew 500 ppm
Sep 24 07:06:20 dist-test-slave-2p7j krb5kdc[11408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233580, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:20.743579 11526 init.cc:377] Logged in from keytab as oryx/127.3.217.1@KRBTEST.COM (short username oryx)
I20260924 07:06:20.744423 11526 webserver.cc:460] Webserver started at http://127.3.217.1:42155/ using document root <none> and password file <none>
I20260924 07:06:20.744940 11526 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:20.745003 11526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:20.745177 11526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:20.747112 11526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "0d49625a33fb4f1b818f190bf9b4ad9f"
format_stamp: "Formatted at 2026-09-24 07:06:20 on dist-test-slave-2p7j"
server_key: "a3a4cf56940a4c2d2f82f22e3d582c7e"
server_key_iv: "10cd1f62016917841d1b02a49e649d71"
server_key_version: "encryptionkey@0"
I20260924 07:06:20.747655 11526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "0d49625a33fb4f1b818f190bf9b4ad9f"
format_stamp: "Formatted at 2026-09-24 07:06:20 on dist-test-slave-2p7j"
server_key: "a3a4cf56940a4c2d2f82f22e3d582c7e"
server_key_iv: "10cd1f62016917841d1b02a49e649d71"
server_key_version: "encryptionkey@0"
I20260924 07:06:20.751616 11526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:06:20.754441 11541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:20.756060 11526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:06:20.756176 11526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "0d49625a33fb4f1b818f190bf9b4ad9f"
format_stamp: "Formatted at 2026-09-24 07:06:20 on dist-test-slave-2p7j"
server_key: "a3a4cf56940a4c2d2f82f22e3d582c7e"
server_key_iv: "10cd1f62016917841d1b02a49e649d71"
server_key_version: "encryptionkey@0"
I20260924 07:06:20.756270 11526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:20.778399 11526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:20.781294 11526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:20.781463 11526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:20.782801 11526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:20.782850 11526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:20.782902 11526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:20.782932 11526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:06:20 dist-test-slave-2p7j krb5kdc[11408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233580, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM
I20260924 07:06:20.792905 11526 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:33363
I20260924 07:06:20.792949 11656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:33363 every 8 connection(s)
I20260924 07:06:20.793949 11526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:06:20.796283 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:20.783610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:49529 (local address 127.3.217.62:43511)
0924 07:06:20.783840 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:20.783851 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:20.784730 (+ 879us) server_negotiation.cc:407] Connection header received
0924 07:06:20.785923 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:20.785927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:20.785983 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:20.786070 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:20.787231 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:20.788024 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:20.789619 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:20.789931 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:20.790026 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:20.792703 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:20.792706 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:20.792709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:20.792734 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:20.794386 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:20.794947 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:20.794950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:20.794951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:20.795021 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:20.795330 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:20.795333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:20.795335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:20.795574 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:20.795670 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:20.796065 (+ 395us) server_negotiation.cc:299] Negotiation successful
0924 07:06:20.796170 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:06:20.796962 11550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:20.784084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:43511 (local address 127.3.217.1:49529)
0924 07:06:20.784555 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:20.784609 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:20.785669 (+ 1060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:20.786260 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:20.786269 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:20.786604 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:20.787087 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:20.787100 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:20.788144 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:20.788148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:20.789512 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:20.789523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:20.790082 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:20.790087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:20.790276 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:20.790845 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:20.790865 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:20.792539 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:20.794522 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:20.794527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:20.794540 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:20.794851 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:20.795100 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:20.795102 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:20.795103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:20.795222 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:20.795658 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:20.795662 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:20.795860 (+ 198us) client_negotiation.cc:769] Sending connection context
0924 07:06:20.796014 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 07:06:20.796231 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":98,"threads_started":1}
I20260924 07:06:20.798076 11657 heartbeater.cc:347] Connected to a master server at 127.3.217.62:43511
I20260924 07:06:20.798210 11657 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:20.798775 11657 heartbeater.cc:511] Master 127.3.217.62:43511 requested a full tablet report, sending...
I20260924 07:06:20.800402 11452 ts_manager.cc:194] Registered new tserver with Master: 0d49625a33fb4f1b818f190bf9b4ad9f (127.3.217.1:33363)
I20260924 07:06:20.800490 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 11526
I20260924 07:06:20.800603 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:06:20.800882 3940 external_mini_cluster.cc:1428] Setting key 898ee57cbe20660705a8d80417720654
I20260924 07:06:20.801874 11452 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.217.1@KRBTEST.COM'} at 127.3.217.1:49529
I20260924 07:06:20.807343 11663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:20.800251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:43511 (local address 127.3.217.1:54055)
0924 07:06:20.800561 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:20.800574 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:20.800669 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:20.800895 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:20.800898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:20.801098 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:20.801207 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:20.801214 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:20.802086 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:20.802089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:20.802847 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:20.802855 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:20.803260 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:20.803263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:20.803346 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:20.803815 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:20.803831 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:20.804215 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:20.805943 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:20.805946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:20.805947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:20.806135 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:20.806423 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:20.806425 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:20.806426 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:20.806531 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:20.806941 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:20.806943 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:20.807012 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 07:06:20.807087 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 07:06:20.807197 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":97,"threads_started":1}
I20260924 07:06:20.807430 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:20.800312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:54055 (local address 127.3.217.62:43511)
0924 07:06:20.800410 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:20.800414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:20.800654 (+ 240us) server_negotiation.cc:407] Connection header received
0924 07:06:20.800742 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:20.800745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:20.800788 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:20.800864 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:20.801308 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:20.801938 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:20.802954 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:20.803158 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:20.803239 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:20.804358 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:20.804361 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:20.804363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:20.804383 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:20.805804 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:20.806236 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:20.806239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:20.806241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:20.806321 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:20.806630 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:20.806634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:20.806635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:20.806832 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:20.806931 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:20.807167 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 07:06:20.807279 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
WARNING: no policy specified for oryx/127.3.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.2.keytab.
Entry for principal oryx/127.3.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.2.keytab.
I20260924 07:06:20.832952 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:43511
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.2.keytab
--principal=oryx/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:32867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:20.950839 11666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:20.951215 11666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:20.951269 11666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:20.956454 11666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:20.956537 11666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:20.956583 11666 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:06:20.956627 11666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:06:20.962082 11666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:32867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.2.keytab
--principal=oryx/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.217.62:43511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:20.963016 11666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:20.964011 11666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:20.973529 11672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:20.973523 11671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:20.973508 11674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:20.973834 11666 server_base.cc:1034] running on GCE node
I20260924 07:06:20.974440 11666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:20.975258 11666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:20.976471 11666 hybrid_clock.cc:648] HybridClock initialized: now 1790233580976406 us; error 74 us; skew 500 ppm
Sep 24 07:06:20 dist-test-slave-2p7j krb5kdc[11408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233580, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:20.979219 11666 init.cc:377] Logged in from keytab as oryx/127.3.217.2@KRBTEST.COM (short username oryx)
I20260924 07:06:20.980118 11666 webserver.cc:460] Webserver started at http://127.3.217.2:39075/ using document root <none> and password file <none>
I20260924 07:06:20.980619 11666 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:20.980682 11666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:20.980855 11666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:20.982729 11666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "28dcee2af0984fb580286b5f85f6ee6e"
format_stamp: "Formatted at 2026-09-24 07:06:20 on dist-test-slave-2p7j"
server_key: "7758acb98a52672c41d810ecdaac971b"
server_key_iv: "82c9da2e2cf3a899ae19a6910ef07c7e"
server_key_version: "encryptionkey@0"
I20260924 07:06:20.983225 11666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "28dcee2af0984fb580286b5f85f6ee6e"
format_stamp: "Formatted at 2026-09-24 07:06:20 on dist-test-slave-2p7j"
server_key: "7758acb98a52672c41d810ecdaac971b"
server_key_iv: "82c9da2e2cf3a899ae19a6910ef07c7e"
server_key_version: "encryptionkey@0"
I20260924 07:06:20.986971 11666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:06:20.989588 11681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:20.990926 11666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:06:20.991025 11666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "28dcee2af0984fb580286b5f85f6ee6e"
format_stamp: "Formatted at 2026-09-24 07:06:20 on dist-test-slave-2p7j"
server_key: "7758acb98a52672c41d810ecdaac971b"
server_key_iv: "82c9da2e2cf3a899ae19a6910ef07c7e"
server_key_version: "encryptionkey@0"
I20260924 07:06:20.991125 11666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:21.007339 11666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:21.011533 11666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:21.011718 11666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:21.013566 11666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:21.013624 11666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:21.013669 11666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:21.013701 11666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:21.024053 11666 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:43527
I20260924 07:06:21.024245 11796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:43527 every 8 connection(s)
Sep 24 07:06:21 dist-test-slave-2p7j krb5kdc[11408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233580, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM
I20260924 07:06:21.025204 11666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:06:21.029639 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.014489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:44901 (local address 127.3.217.62:43511)
0924 07:06:21.014633 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:21.014638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:21.015282 (+ 644us) server_negotiation.cc:407] Connection header received
0924 07:06:21.016463 (+ 1181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:21.016466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:21.016516 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:21.016571 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:21.019107 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.019951 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.021710 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.022031 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.022142 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.025417 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:21.025421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:21.025424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:21.025451 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:21.026817 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:21.027426 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:21.027432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:21.027433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:21.027699 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:21.028300 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:21.028304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:21.028305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:21.028561 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:21.028711 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:21.029378 (+ 667us) server_negotiation.cc:299] Negotiation successful
0924 07:06:21.029516 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 07:06:21.029991 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 11666
I20260924 07:06:21.030112 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:06:21.030417 3940 external_mini_cluster.cc:1428] Setting key 5d728693a0784d066bf23ac6f086bd31
I20260924 07:06:21.030611 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.014684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:43511 (local address 127.3.217.2:44901)
0924 07:06:21.015106 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:21.015164 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:21.016124 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:21.016941 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:21.016949 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:21.017819 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:21.018945 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.018972 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.020147 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.020152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.021572 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.021585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.022341 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.022346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.022582 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.023220 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:21.023239 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:21.025211 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:21.026944 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:21.026948 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:21.026959 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:21.027218 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:21.027855 (+ 637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:21.027858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:21.027859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:21.028155 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:21.028733 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:21.028738 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:21.029113 (+ 375us) client_negotiation.cc:769] Sending connection context
0924 07:06:21.029279 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 07:06:21.029546 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":94,"threads_started":1}
I20260924 07:06:21.032061 11797 heartbeater.cc:347] Connected to a master server at 127.3.217.62:43511
I20260924 07:06:21.032202 11797 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:21.032648 11797 heartbeater.cc:511] Master 127.3.217.62:43511 requested a full tablet report, sending...
I20260924 07:06:21.033579 11452 ts_manager.cc:194] Registered new tserver with Master: 28dcee2af0984fb580286b5f85f6ee6e (127.3.217.2:43527)
I20260924 07:06:21.034160 11452 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.217.2@KRBTEST.COM'} at 127.3.217.2:44901
WARNING: no policy specified for oryx/127.3.217.3@KRBTEST.COM; defaulting to no policy
I20260924 07:06:21.043164 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.035641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:44079 (local address 127.3.217.62:43511)
0924 07:06:21.035817 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:21.035823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:21.036116 (+ 293us) server_negotiation.cc:407] Connection header received
0924 07:06:21.036186 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:21.036189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:21.036242 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:21.036331 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:21.036896 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.037447 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.038409 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.038732 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.038864 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.040345 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:21.040350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:21.040352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:21.040375 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:21.041626 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:21.042020 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:21.042023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:21.042024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:21.042090 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:21.042335 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:21.042338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:21.042340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:21.042556 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:21.042640 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:21.042892 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 07:06:21.043035 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 07:06:21.043169 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.035591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:43511 (local address 127.3.217.2:44079)
0924 07:06:21.035939 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:21.035955 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:21.036063 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:21.036384 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:21.036387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:21.036610 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:21.036720 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.036727 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.037615 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.037618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.038182 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.038191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.038893 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.038898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.039014 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.039828 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:21.039840 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:21.040184 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:21.041737 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:21.041741 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:21.041743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:21.041928 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:21.042175 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:21.042178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:21.042180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:21.042252 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:21.042654 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:21.042657 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:21.042740 (+ 83us) client_negotiation.cc:769] Sending connection context
0924 07:06:21.042824 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 07:06:21.042971 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":126,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.3.keytab.
Entry for principal oryx/127.3.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.3.keytab.
I20260924 07:06:21.065316 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:43511
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.3.keytab
--principal=oryx/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:32867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:21.183205 11806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:21.183447 11806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:21.183564 11806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:21.188293 11806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:21.188364 11806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:21.188421 11806 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:06:21.188476 11806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:06:21.193853 11806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:32867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.3.keytab
--principal=oryx/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.217.62:43511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:21.194811 11806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:21.195972 11806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:21.206969 11814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:21.207170 11811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:21.207209 11806 server_base.cc:1034] running on GCE node
W20260924 07:06:21.207291 11812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:21.207791 11806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:21.208438 11806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:21.209692 11806 hybrid_clock.cc:648] HybridClock initialized: now 1790233581209651 us; error 31 us; skew 500 ppm
Sep 24 07:06:21 dist-test-slave-2p7j krb5kdc[11408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233581, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:21.212618 11806 init.cc:377] Logged in from keytab as oryx/127.3.217.3@KRBTEST.COM (short username oryx)
I20260924 07:06:21.213405 11806 webserver.cc:460] Webserver started at http://127.3.217.3:34635/ using document root <none> and password file <none>
I20260924 07:06:21.213912 11806 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:21.213975 11806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:21.214150 11806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:21.216466 11806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "725254e86479418ab43dddee8a117821"
format_stamp: "Formatted at 2026-09-24 07:06:21 on dist-test-slave-2p7j"
server_key: "40719a8450e80811b3a3a5e7997e67d3"
server_key_iv: "c3c6ad87ecf26e761c7f35967435e09a"
server_key_version: "encryptionkey@0"
I20260924 07:06:21.216998 11806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "725254e86479418ab43dddee8a117821"
format_stamp: "Formatted at 2026-09-24 07:06:21 on dist-test-slave-2p7j"
server_key: "40719a8450e80811b3a3a5e7997e67d3"
server_key_iv: "c3c6ad87ecf26e761c7f35967435e09a"
server_key_version: "encryptionkey@0"
I20260924 07:06:21.220870 11806 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:06:21.224097 11821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:21.225427 11806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:06:21.225543 11806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "725254e86479418ab43dddee8a117821"
format_stamp: "Formatted at 2026-09-24 07:06:21 on dist-test-slave-2p7j"
server_key: "40719a8450e80811b3a3a5e7997e67d3"
server_key_iv: "c3c6ad87ecf26e761c7f35967435e09a"
server_key_version: "encryptionkey@0"
I20260924 07:06:21.225646 11806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:21.269393 11806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:21.273471 11806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:21.273669 11806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:21.276286 11806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:21.276374 11806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:21.276434 11806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:21.276492 11806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 24 07:06:21 dist-test-slave-2p7j krb5kdc[11408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233581, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM
I20260924 07:06:21.287994 11806 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:43785
I20260924 07:06:21.288074 11937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:43785 every 8 connection(s)
I20260924 07:06:21.289088 11806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:06:21.291258 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.276539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:51185 (local address 127.3.217.62:43511)
0924 07:06:21.276676 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:21.276681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:21.277786 (+ 1105us) server_negotiation.cc:407] Connection header received
0924 07:06:21.278994 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:21.278998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:21.279053 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:21.279106 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:21.280812 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.281817 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.283125 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.283412 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.283548 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.287077 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:21.287082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:21.287084 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:21.287113 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:21.288820 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:21.289401 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:21.289405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:21.289407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:21.289490 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:21.289899 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:21.289902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:21.289904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:21.290125 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:21.290224 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:21.291026 (+ 802us) server_negotiation.cc:299] Negotiation successful
0924 07:06:21.291135 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:06:21.292531 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.277146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:43511 (local address 127.3.217.3:51185)
0924 07:06:21.277619 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:21.277666 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:21.278757 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:21.279534 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:21.279543 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:21.279900 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:21.280475 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.280494 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.281960 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.281963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.282991 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.283001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.283626 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.283631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.283813 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.284439 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:21.284461 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:21.286854 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:21.288963 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:21.288967 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:21.288979 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:21.289277 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:21.289605 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:21.289608 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:21.289610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:21.289782 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:21.290277 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:21.290282 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:21.290532 (+ 250us) client_negotiation.cc:769] Sending connection context
0924 07:06:21.290929 (+ 397us) client_negotiation.cc:240] Negotiation successful
0924 07:06:21.291429 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":103,"threads_started":1}
I20260924 07:06:21.293913 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 11806
I20260924 07:06:21.294064 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:06:21.294068 11938 heartbeater.cc:347] Connected to a master server at 127.3.217.62:43511
I20260924 07:06:21.294342 11938 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:21.294443 3940 external_mini_cluster.cc:1428] Setting key 6a5bb0ae7ac2223b99898fcdb3544df9
I20260924 07:06:21.295378 11938 heartbeater.cc:511] Master 127.3.217.62:43511 requested a full tablet report, sending...
I20260924 07:06:21.296759 11452 ts_manager.cc:194] Registered new tserver with Master: 725254e86479418ab43dddee8a117821 (127.3.217.3:43785)
I20260924 07:06:21.297575 11452 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.217.3@KRBTEST.COM'} at 127.3.217.3:51185
I20260924 07:06:21.304631 11944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.296360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:43511 (local address 127.3.217.3:52843)
0924 07:06:21.296623 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:21.296637 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:21.296725 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:21.297009 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:21.297011 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:21.297218 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:21.297342 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.297349 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.298496 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.298499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.299217 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.299225 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.299919 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.299922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.300004 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.300686 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:21.300697 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:21.301027 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:21.302788 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:21.302791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:21.302793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:21.303079 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:21.303543 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:21.303548 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:21.303550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:21.303659 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:21.304165 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:21.304167 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:21.304249 (+ 82us) client_negotiation.cc:769] Sending connection context
0924 07:06:21.304348 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 07:06:21.304476 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":98,"threads_started":1}
I20260924 07:06:21.304906 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.296230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:52843 (local address 127.3.217.62:43511)
0924 07:06:21.296352 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:21.296356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:21.296706 (+ 350us) server_negotiation.cc:407] Connection header received
0924 07:06:21.296796 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:21.296799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:21.296838 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:21.296907 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:21.297574 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.298296 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.299353 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.299755 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.299886 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.301197 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:21.301200 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:21.301202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:21.301225 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:21.302617 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:21.303210 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:21.303213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:21.303214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:21.303289 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:21.303784 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:21.303786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:21.303788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:21.304059 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:21.304145 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:21.304502 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 07:06:21.304708 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:06:21.309583 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 07:06:21 dist-test-slave-2p7j krb5kdc[11408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233581, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 07:06:21 dist-test-slave-2p7j krb5kdc[11408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:21 dist-test-slave-2p7j krb5kdc[11408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 07:06:21.335260 11503 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:21.329238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54396 (local address 127.3.217.62:43511)
0924 07:06:21.329361 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:21.329366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:21.329380 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:06:21.329424 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:21.329426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:21.329463 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:21.329530 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:21.330235 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.330760 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.331570 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.331793 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.331885 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.335133 (+ 3248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54396: BlockingRecv error: recv got EOF from 127.0.0.1:54396 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 07:06:21 dist-test-slave-2p7j krb5kdc[11408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233581, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM
I20260924 07:06:21.344982 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.336604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54400 (local address 127.3.217.62:43511)
0924 07:06:21.336700 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:21.336703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:21.336823 (+ 120us) server_negotiation.cc:407] Connection header received
0924 07:06:21.336904 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:21.336906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:21.336946 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:21.337010 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:21.338012 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.338536 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.339440 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.339693 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.339775 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.342028 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:21.342031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:21.342033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:21.342053 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:21.343235 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:21.343951 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:21.343954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:21.343955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:21.344032 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:21.344302 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:21.344305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:21.344306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:21.344483 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:21.344591 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:21.344788 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 07:06:21.344873 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:06:21.348531 11452 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:06:21.351420 11452 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:06:21.366283 11962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.360409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.3.217.2:43527)
0924 07:06:21.361716 (+ 1307us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:21.361726 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:21.361744 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:06:21.361806 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:21.361811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:21.361959 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:21.362182 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:21.362395 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.363468 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.364997 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.365534 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.365890 (+ 356us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.366041 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:21.366090 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 07:06:21.366168 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1222,"thread_start_us":140,"threads_started":1}
I20260924 07:06:21.366276 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.360873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:43785 (local address 127.0.0.1:54942)
0924 07:06:21.361169 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:21.361181 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:21.361265 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:21.361816 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:21.361822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:21.361842 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:21.362002 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.362010 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.363569 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.363572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.364681 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.364689 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.365664 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.365681 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.365815 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.365883 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 07:06:21.365985 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 07:06:21.366124 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":3,"thread_start_us":181,"threads_started":1}
I20260924 07:06:21.366338 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.361300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44384 (local address 127.3.217.1:33363)
0924 07:06:21.361551 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:21.361557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:21.361579 (+ 22us) server_negotiation.cc:407] Connection header received
0924 07:06:21.361649 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:21.361654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:21.361820 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:21.361966 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:21.362368 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.363077 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.364707 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.365495 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.365603 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.365736 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:21.366138 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 07:06:21.366228 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":99,"threads_started":1}
I20260924 07:06:21.366513 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.360302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:33363 (local address 127.0.0.1:44384)
0924 07:06:21.360875 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:21.360890 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:21.360988 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:21.362091 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:21.362094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:21.362124 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:21.362247 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.362254 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.363199 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.363202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.364533 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.364544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.365858 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.365861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.365962 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.365984 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:06:21.366049 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 07:06:21.366336 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":223,"spinlock_wait_cycles":7680,"thread_start_us":117,"threads_started":1}
I20260924 07:06:21.366285 11961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.360237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:43527 (local address 127.0.0.1:58986)
0924 07:06:21.360659 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:21.360703 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:21.360821 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:21.362091 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:21.362094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:21.362118 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:21.362273 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.362280 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.363604 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.363608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.364829 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.364838 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.365664 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.365668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.365821 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.365922 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 07:06:21.365985 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 07:06:21.366140 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":142,"spinlock_wait_cycles":1792,"thread_start_us":67,"threads_started":1}
I20260924 07:06:21.367105 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.361061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54942 (local address 127.3.217.3:43785)
0924 07:06:21.361367 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:21.361376 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:21.361394 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:06:21.361460 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:21.361466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:21.361659 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:21.362140 (+ 481us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:21.362261 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.363406 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.364793 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.365504 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.365603 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.365752 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:21.366766 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0924 07:06:21.366969 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":89,"threads_started":1}
I20260924 07:06:21.369495 11591 tablet_service.cc:1467] Processing CreateTablet for tablet 6327b4ede987492e8d9771b7182384e0 (DEFAULT_TABLE table=test-table [id=ffd48cf50f11493a89103ec220929e9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:21.369477 11731 tablet_service.cc:1467] Processing CreateTablet for tablet 6327b4ede987492e8d9771b7182384e0 (DEFAULT_TABLE table=test-table [id=ffd48cf50f11493a89103ec220929e9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:21.370056 11871 tablet_service.cc:1467] Processing CreateTablet for tablet 6327b4ede987492e8d9771b7182384e0 (DEFAULT_TABLE table=test-table [id=ffd48cf50f11493a89103ec220929e9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:21.370594 11591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6327b4ede987492e8d9771b7182384e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:21.370594 11731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6327b4ede987492e8d9771b7182384e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:21.370868 11871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6327b4ede987492e8d9771b7182384e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:21.377421 11966 tablet_bootstrap.cc:492] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821: Bootstrap starting.
I20260924 07:06:21.377907 11968 tablet_bootstrap.cc:492] T 6327b4ede987492e8d9771b7182384e0 P 28dcee2af0984fb580286b5f85f6ee6e: Bootstrap starting.
I20260924 07:06:21.379303 11966 tablet_bootstrap.cc:654] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:21.379608 11968 tablet_bootstrap.cc:654] T 6327b4ede987492e8d9771b7182384e0 P 28dcee2af0984fb580286b5f85f6ee6e: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:21.379832 11967 tablet_bootstrap.cc:492] T 6327b4ede987492e8d9771b7182384e0 P 0d49625a33fb4f1b818f190bf9b4ad9f: Bootstrap starting.
I20260924 07:06:21.380182 11966 log.cc:824] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:21.380368 11968 log.cc:824] T 6327b4ede987492e8d9771b7182384e0 P 28dcee2af0984fb580286b5f85f6ee6e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:21.381630 11967 tablet_bootstrap.cc:654] T 6327b4ede987492e8d9771b7182384e0 P 0d49625a33fb4f1b818f190bf9b4ad9f: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:21.382134 11966 tablet_bootstrap.cc:492] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821: No bootstrap required, opened a new log
I20260924 07:06:21.382318 11966 ts_tablet_manager.cc:1397] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:06:21.382366 11967 log.cc:824] T 6327b4ede987492e8d9771b7182384e0 P 0d49625a33fb4f1b818f190bf9b4ad9f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:21.382920 11968 tablet_bootstrap.cc:492] T 6327b4ede987492e8d9771b7182384e0 P 28dcee2af0984fb580286b5f85f6ee6e: No bootstrap required, opened a new log
I20260924 07:06:21.383148 11968 ts_tablet_manager.cc:1397] T 6327b4ede987492e8d9771b7182384e0 P 28dcee2af0984fb580286b5f85f6ee6e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260924 07:06:21.384176 11967 tablet_bootstrap.cc:492] T 6327b4ede987492e8d9771b7182384e0 P 0d49625a33fb4f1b818f190bf9b4ad9f: No bootstrap required, opened a new log
I20260924 07:06:21.384341 11967 ts_tablet_manager.cc:1397] T 6327b4ede987492e8d9771b7182384e0 P 0d49625a33fb4f1b818f190bf9b4ad9f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:06:21.385414 11966 raft_consensus.cc:348] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.385619 11966 raft_consensus.cc:374] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:21.385689 11966 raft_consensus.cc:729] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 725254e86479418ab43dddee8a117821, State: Initialized, Role: FOLLOWER
I20260924 07:06:21.386070 11966 consensus_queue.cc:260] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.386641 11938 heartbeater.cc:503] Master 127.3.217.62:43511 was elected leader, sending a full tablet report...
I20260924 07:06:21.387022 11966 ts_tablet_manager.cc:1428] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:06:21.387637 11968 raft_consensus.cc:348] T 6327b4ede987492e8d9771b7182384e0 P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.388005 11968 raft_consensus.cc:374] T 6327b4ede987492e8d9771b7182384e0 P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:21.388099 11968 raft_consensus.cc:729] T 6327b4ede987492e8d9771b7182384e0 P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28dcee2af0984fb580286b5f85f6ee6e, State: Initialized, Role: FOLLOWER
I20260924 07:06:21.388530 11968 consensus_queue.cc:260] T 6327b4ede987492e8d9771b7182384e0 P 28dcee2af0984fb580286b5f85f6ee6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.389163 11797 heartbeater.cc:503] Master 127.3.217.62:43511 was elected leader, sending a full tablet report...
I20260924 07:06:21.389467 11968 ts_tablet_manager.cc:1428] T 6327b4ede987492e8d9771b7182384e0 P 28dcee2af0984fb580286b5f85f6ee6e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 07:06:21.389391 11967 raft_consensus.cc:348] T 6327b4ede987492e8d9771b7182384e0 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.389611 11967 raft_consensus.cc:374] T 6327b4ede987492e8d9771b7182384e0 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:21.389663 11967 raft_consensus.cc:729] T 6327b4ede987492e8d9771b7182384e0 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d49625a33fb4f1b818f190bf9b4ad9f, State: Initialized, Role: FOLLOWER
I20260924 07:06:21.390093 11967 consensus_queue.cc:260] T 6327b4ede987492e8d9771b7182384e0 P 0d49625a33fb4f1b818f190bf9b4ad9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.390973 11967 ts_tablet_manager.cc:1428] T 6327b4ede987492e8d9771b7182384e0 P 0d49625a33fb4f1b818f190bf9b4ad9f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 07:06:21.391124 11657 heartbeater.cc:503] Master 127.3.217.62:43511 was elected leader, sending a full tablet report...
I20260924 07:06:21.405323 11972 raft_consensus.cc:482] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:06:21.405473 11972 raft_consensus.cc:504] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.406504 11972 leader_election.cc:288] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28dcee2af0984fb580286b5f85f6ee6e (127.3.217.2:43527), 0d49625a33fb4f1b818f190bf9b4ad9f (127.3.217.1:33363)
I20260924 07:06:21.410359 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.406724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:43527 (local address 127.3.217.3:45463)
0924 07:06:21.406946 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:21.406971 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:21.407063 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:21.407287 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:21.407289 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:21.407305 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:21.407410 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.407415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.408132 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.408135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.408975 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.408985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.409986 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.409989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.410100 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.410128 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 07:06:21.410197 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 07:06:21.410260 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260924 07:06:21.410442 11962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.406883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:45463 (local address 127.3.217.2:43527)
0924 07:06:21.406994 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:21.406998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:21.407014 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:06:21.407163 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:21.407166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:21.407203 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:21.407260 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:21.407532 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.408023 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.409111 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.409714 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.410047 (+ 333us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.410210 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:21.410255 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 07:06:21.410325 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:06:21.411103 11751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6327b4ede987492e8d9771b7182384e0" candidate_uuid: "725254e86479418ab43dddee8a117821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dcee2af0984fb580286b5f85f6ee6e" is_pre_election: true
I20260924 07:06:21.411327 11975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.406735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:33363 (local address 127.3.217.3:45765)
0924 07:06:21.407149 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:21.407162 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:21.407253 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:21.407486 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:21.407489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:21.407524 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:21.407657 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.407662 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.408338 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.408341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.409765 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.409774 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.410939 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.410942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.411034 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.411056 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:06:21.411128 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 07:06:21.411193 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":161,"thread_start_us":117,"threads_started":1}
I20260924 07:06:21.411334 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.406893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:45765 (local address 127.3.217.1:33363)
0924 07:06:21.407006 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:21.407010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:21.407243 (+ 233us) server_negotiation.cc:407] Connection header received
0924 07:06:21.407365 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:21.407367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:21.407405 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:21.407463 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:21.407757 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.408234 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.410098 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.410790 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.410895 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.410946 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:21.411157 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 07:06:21.411216 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:06:21.411427 11751 raft_consensus.cc:2454] T 6327b4ede987492e8d9771b7182384e0 P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 725254e86479418ab43dddee8a117821 in term 0.
I20260924 07:06:21.412081 11825 leader_election.cc:302] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28dcee2af0984fb580286b5f85f6ee6e, 725254e86479418ab43dddee8a117821; no voters:
I20260924 07:06:21.412169 11609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6327b4ede987492e8d9771b7182384e0" candidate_uuid: "725254e86479418ab43dddee8a117821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" is_pre_election: true
I20260924 07:06:21.412349 11972 raft_consensus.cc:2781] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:06:21.412429 11972 raft_consensus.cc:482] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:06:21.412428 11609 raft_consensus.cc:2454] T 6327b4ede987492e8d9771b7182384e0 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 725254e86479418ab43dddee8a117821 in term 0.
I20260924 07:06:21.412477 11972 raft_consensus.cc:3037] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:21.413514 11972 raft_consensus.cc:504] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.413793 11972 leader_election.cc:288] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821 [CANDIDATE]: Term 1 election: Requested vote from peers 28dcee2af0984fb580286b5f85f6ee6e (127.3.217.2:43527), 0d49625a33fb4f1b818f190bf9b4ad9f (127.3.217.1:33363)
I20260924 07:06:21.414108 11751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6327b4ede987492e8d9771b7182384e0" candidate_uuid: "725254e86479418ab43dddee8a117821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dcee2af0984fb580286b5f85f6ee6e"
I20260924 07:06:21.414116 11609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6327b4ede987492e8d9771b7182384e0" candidate_uuid: "725254e86479418ab43dddee8a117821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f"
I20260924 07:06:21.414209 11609 raft_consensus.cc:3037] T 6327b4ede987492e8d9771b7182384e0 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:21.414217 11751 raft_consensus.cc:3037] T 6327b4ede987492e8d9771b7182384e0 P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:21.415148 11609 raft_consensus.cc:2454] T 6327b4ede987492e8d9771b7182384e0 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 725254e86479418ab43dddee8a117821 in term 1.
I20260924 07:06:21.415467 11822 leader_election.cc:302] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d49625a33fb4f1b818f190bf9b4ad9f, 725254e86479418ab43dddee8a117821; no voters:
I20260924 07:06:21.415650 11972 raft_consensus.cc:2781] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:21.415759 11751 raft_consensus.cc:2454] T 6327b4ede987492e8d9771b7182384e0 P 28dcee2af0984fb580286b5f85f6ee6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 725254e86479418ab43dddee8a117821 in term 1.
I20260924 07:06:21.415838 11972 raft_consensus.cc:686] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821 [term 1 LEADER]: Becoming Leader. State: Replica: 725254e86479418ab43dddee8a117821, State: Running, Role: LEADER
I20260924 07:06:21.416189 11972 consensus_queue.cc:237] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.419400 11451 catalog_manager.cc:5495] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 725254e86479418ab43dddee8a117821 (127.3.217.3). New cstate: current_term: 1 leader_uuid: "725254e86479418ab43dddee8a117821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } health_report { overall_health: HEALTHY } } }
I20260924 07:06:21.432188 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.428245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54956 (local address 127.3.217.3:43785)
0924 07:06:21.428466 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:21.428470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:21.428493 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:06:21.428621 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:21.428623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:21.428692 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:21.428818 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:06:21.429124 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.429858 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.430751 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.430978 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.431065 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.431377 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:06:21.431811 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:06:21.431922 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:21.432034 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 07:06:21.432092 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 07:06:21.437415 11751 raft_consensus.cc:1261] T 6327b4ede987492e8d9771b7182384e0 P 28dcee2af0984fb580286b5f85f6ee6e [term 1 FOLLOWER]: Refusing update from remote peer 725254e86479418ab43dddee8a117821: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:06:21.437422 11609 raft_consensus.cc:1261] T 6327b4ede987492e8d9771b7182384e0 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 1 FOLLOWER]: Refusing update from remote peer 725254e86479418ab43dddee8a117821: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:06:21.438019 11972 consensus_queue.cc:1035] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:06:21.438416 11976 consensus_queue.cc:1035] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:06:21.447217 11982 mvcc.cc:205] Tried to move back new op lower bound from 7332796749561757696 to 7332796749482983424. Current Snapshot: MvccSnapshot[applied={T|T < 7332796749561757696}]
I20260924 07:06:21.448371 11980 mvcc.cc:205] Tried to move back new op lower bound from 7332796749561757696 to 7332796749482983424. Current Snapshot: MvccSnapshot[applied={T|T < 7332796749561757696 or (T in {7332796749561757696})}]
I20260924 07:06:21.454926 11452 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:06:21.455076 11452 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:06:21.457366 11452 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040: Sending DeleteTablet for 3 replicas of tablet 6327b4ede987492e8d9771b7182384e0
I20260924 07:06:21.458083 11591 tablet_service.cc:1514] Processing DeleteTablet for tablet 6327b4ede987492e8d9771b7182384e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:21 UTC) from {username='oryx', principal='oryx/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:44384
I20260924 07:06:21.458086 11731 tablet_service.cc:1514] Processing DeleteTablet for tablet 6327b4ede987492e8d9771b7182384e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:21 UTC) from {username='oryx', principal='oryx/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:58986
I20260924 07:06:21.458539 11989 tablet_replica.cc:330] T 6327b4ede987492e8d9771b7182384e0 P 28dcee2af0984fb580286b5f85f6ee6e: stopping tablet replica
I20260924 07:06:21.458567 11452 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 07:06:21.458688 11871 tablet_service.cc:1514] Processing DeleteTablet for tablet 6327b4ede987492e8d9771b7182384e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:21 UTC) from {username='oryx', principal='oryx/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:54942
I20260924 07:06:21.458945 11989 raft_consensus.cc:2229] T 6327b4ede987492e8d9771b7182384e0 P 28dcee2af0984fb580286b5f85f6ee6e [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 07:06:21.458974 11452 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:06:21.459143 11988 tablet_replica.cc:330] T 6327b4ede987492e8d9771b7182384e0 P 0d49625a33fb4f1b818f190bf9b4ad9f: stopping tablet replica
I20260924 07:06:21.459224 11989 raft_consensus.cc:2258] T 6327b4ede987492e8d9771b7182384e0 P 28dcee2af0984fb580286b5f85f6ee6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:21.459440 11988 raft_consensus.cc:2229] T 6327b4ede987492e8d9771b7182384e0 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:06:21.459720 11988 raft_consensus.cc:2258] T 6327b4ede987492e8d9771b7182384e0 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:21.460143 11990 tablet_replica.cc:330] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821: stopping tablet replica
I20260924 07:06:21.460713 11990 raft_consensus.cc:2229] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:06:21.461135 11988 ts_tablet_manager.cc:1905] T 6327b4ede987492e8d9771b7182384e0 P 0d49625a33fb4f1b818f190bf9b4ad9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:06:21.461493 11990 raft_consensus.cc:2258] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:21.463477 11989 ts_tablet_manager.cc:1905] T 6327b4ede987492e8d9771b7182384e0 P 28dcee2af0984fb580286b5f85f6ee6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:06:21.463928 11871 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4ee02a7172444b9cd64c46f2c82f26 (DEFAULT_TABLE table=test-table [id=b9daafacae7c4ff4aa816ccb2400d42a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:21.464186 11871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4ee02a7172444b9cd64c46f2c82f26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:21.464334 11988 ts_tablet_manager.cc:1918] T 6327b4ede987492e8d9771b7182384e0 P 0d49625a33fb4f1b818f190bf9b4ad9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:06:21.464527 11988 log.cc:1196] T 6327b4ede987492e8d9771b7182384e0 P 0d49625a33fb4f1b818f190bf9b4ad9f: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-0/wal/wals/6327b4ede987492e8d9771b7182384e0
I20260924 07:06:21.464962 11988 ts_tablet_manager.cc:1939] T 6327b4ede987492e8d9771b7182384e0 P 0d49625a33fb4f1b818f190bf9b4ad9f: Deleting consensus metadata
I20260924 07:06:21.466784 11966 tablet_bootstrap.cc:492] T 7e4ee02a7172444b9cd64c46f2c82f26 P 725254e86479418ab43dddee8a117821: Bootstrap starting.
I20260924 07:06:21.466857 11989 ts_tablet_manager.cc:1918] T 6327b4ede987492e8d9771b7182384e0 P 28dcee2af0984fb580286b5f85f6ee6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:06:21.467034 11989 log.cc:1196] T 6327b4ede987492e8d9771b7182384e0 P 28dcee2af0984fb580286b5f85f6ee6e: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/6327b4ede987492e8d9771b7182384e0
I20260924 07:06:21.467247 11989 ts_tablet_manager.cc:1939] T 6327b4ede987492e8d9771b7182384e0 P 28dcee2af0984fb580286b5f85f6ee6e: Deleting consensus metadata
I20260924 07:06:21.467705 11966 tablet_bootstrap.cc:654] T 7e4ee02a7172444b9cd64c46f2c82f26 P 725254e86479418ab43dddee8a117821: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:21.468094 11591 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4ee02a7172444b9cd64c46f2c82f26 (DEFAULT_TABLE table=test-table [id=b9daafacae7c4ff4aa816ccb2400d42a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:21.468321 11591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4ee02a7172444b9cd64c46f2c82f26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:21.468812 11966 tablet_bootstrap.cc:492] T 7e4ee02a7172444b9cd64c46f2c82f26 P 725254e86479418ab43dddee8a117821: No bootstrap required, opened a new log
I20260924 07:06:21.468881 11966 ts_tablet_manager.cc:1397] T 7e4ee02a7172444b9cd64c46f2c82f26 P 725254e86479418ab43dddee8a117821: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:06:21.469296 11966 raft_consensus.cc:348] T 7e4ee02a7172444b9cd64c46f2c82f26 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.469391 11966 raft_consensus.cc:374] T 7e4ee02a7172444b9cd64c46f2c82f26 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:21.469430 11966 raft_consensus.cc:729] T 7e4ee02a7172444b9cd64c46f2c82f26 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 725254e86479418ab43dddee8a117821, State: Initialized, Role: FOLLOWER
I20260924 07:06:21.469542 11966 consensus_queue.cc:260] T 7e4ee02a7172444b9cd64c46f2c82f26 P 725254e86479418ab43dddee8a117821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.469770 11966 ts_tablet_manager.cc:1428] T 7e4ee02a7172444b9cd64c46f2c82f26 P 725254e86479418ab43dddee8a117821: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:06:21.470322 11967 tablet_bootstrap.cc:492] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f: Bootstrap starting.
I20260924 07:06:21.470418 11437 catalog_manager.cc:4841] TS 0d49625a33fb4f1b818f190bf9b4ad9f (127.3.217.1:33363): tablet 6327b4ede987492e8d9771b7182384e0 (table test-table [id=ffd48cf50f11493a89103ec220929e9a]) successfully deleted
I20260924 07:06:21.471180 11967 tablet_bootstrap.cc:654] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:21.471170 11440 catalog_manager.cc:4841] TS 28dcee2af0984fb580286b5f85f6ee6e (127.3.217.2:43527): tablet 6327b4ede987492e8d9771b7182384e0 (table test-table [id=ffd48cf50f11493a89103ec220929e9a]) successfully deleted
I20260924 07:06:21.471372 11731 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4ee02a7172444b9cd64c46f2c82f26 (DEFAULT_TABLE table=test-table [id=b9daafacae7c4ff4aa816ccb2400d42a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:21.471632 11731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4ee02a7172444b9cd64c46f2c82f26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:21.472702 11990 ts_tablet_manager.cc:1905] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:06:21.473647 11967 tablet_bootstrap.cc:492] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f: No bootstrap required, opened a new log
I20260924 07:06:21.473714 11968 tablet_bootstrap.cc:492] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e: Bootstrap starting.
I20260924 07:06:21.473716 11967 ts_tablet_manager.cc:1397] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:06:21.474406 11967 raft_consensus.cc:348] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.474516 11967 raft_consensus.cc:374] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:21.474548 11967 raft_consensus.cc:729] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d49625a33fb4f1b818f190bf9b4ad9f, State: Initialized, Role: FOLLOWER
I20260924 07:06:21.474608 11968 tablet_bootstrap.cc:654] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:21.474679 11967 consensus_queue.cc:260] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.474928 11967 ts_tablet_manager.cc:1428] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:06:21.475016 11990 ts_tablet_manager.cc:1918] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:06:21.475106 11990 log.cc:1196] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/6327b4ede987492e8d9771b7182384e0
I20260924 07:06:21.475373 11990 ts_tablet_manager.cc:1939] T 6327b4ede987492e8d9771b7182384e0 P 725254e86479418ab43dddee8a117821: Deleting consensus metadata
I20260924 07:06:21.476092 11968 tablet_bootstrap.cc:492] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e: No bootstrap required, opened a new log
I20260924 07:06:21.476168 11968 ts_tablet_manager.cc:1397] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:06:21.476291 11437 catalog_manager.cc:4841] TS 725254e86479418ab43dddee8a117821 (127.3.217.3:43785): tablet 6327b4ede987492e8d9771b7182384e0 (table test-table [id=ffd48cf50f11493a89103ec220929e9a]) successfully deleted
I20260924 07:06:21.476575 11968 raft_consensus.cc:348] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.476651 11968 raft_consensus.cc:374] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:21.476670 11968 raft_consensus.cc:729] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28dcee2af0984fb580286b5f85f6ee6e, State: Initialized, Role: FOLLOWER
I20260924 07:06:21.476769 11968 consensus_queue.cc:260] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.477002 11968 ts_tablet_manager.cc:1428] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:06:21.495661 11973 raft_consensus.cc:482] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:06:21.495872 11973 raft_consensus.cc:504] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.497262 11973 leader_election.cc:288] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d49625a33fb4f1b818f190bf9b4ad9f (127.3.217.1:33363), 725254e86479418ab43dddee8a117821 (127.3.217.3:43785)
I20260924 07:06:21.501745 11974 raft_consensus.cc:482] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:06:21.501868 11974 raft_consensus.cc:504] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.502988 11974 leader_election.cc:288] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28dcee2af0984fb580286b5f85f6ee6e (127.3.217.2:43527), 725254e86479418ab43dddee8a117821 (127.3.217.3:43785)
I20260924 07:06:21.503028 11994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.497670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:43785 (local address 127.3.217.2:57667)
0924 07:06:21.498105 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:21.498118 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:21.498195 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:21.499292 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:21.499295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:21.499312 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:21.499428 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.499434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.500444 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.500448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.501673 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.501683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.502617 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.502620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.502712 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.502735 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:06:21.502812 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 07:06:21.502881 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":140,"threads_started":1}
I20260924 07:06:21.503054 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.498741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:57667 (local address 127.3.217.3:43785)
0924 07:06:21.498884 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:21.498888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:21.498904 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:06:21.498953 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:21.498956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:21.499000 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:21.499071 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:21.499596 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.500317 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.501806 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.502353 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.502429 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.502508 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:21.502863 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 07:06:21.502934 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 07:06:21.503518 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4ee02a7172444b9cd64c46f2c82f26" candidate_uuid: "28dcee2af0984fb580286b5f85f6ee6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "725254e86479418ab43dddee8a117821" is_pre_election: true
I20260924 07:06:21.503751 11891 raft_consensus.cc:2454] T 7e4ee02a7172444b9cd64c46f2c82f26 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28dcee2af0984fb580286b5f85f6ee6e in term 0.
I20260924 07:06:21.504120 11682 leader_election.cc:302] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28dcee2af0984fb580286b5f85f6ee6e, 725254e86479418ab43dddee8a117821; no voters:
I20260924 07:06:21.504357 11973 raft_consensus.cc:2781] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:06:21.504402 11973 raft_consensus.cc:482] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:06:21.504436 11973 raft_consensus.cc:3037] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:21.505294 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.497446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:33363 (local address 127.3.217.2:47175)
0924 07:06:21.497577 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:21.497592 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:21.497675 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:21.499001 (+ 1326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:21.499004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:21.499022 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:21.499129 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.499135 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.500336 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.500339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.501554 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.501564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.504886 (+ 3322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.504890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.505011 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.505032 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:06:21.505099 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 07:06:21.505165 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":3}
I20260924 07:06:21.505846 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.498399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:47175 (local address 127.3.217.1:33363)
0924 07:06:21.498520 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:21.498524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:21.498537 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:06:21.498616 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:21.498619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:21.498660 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:21.498723 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:21.499268 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.500215 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.501693 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.504748 (+ 3055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.504855 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.504906 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:21.505652 (+ 746us) server_negotiation.cc:299] Negotiation successful
0924 07:06:21.505729 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:06:21.506112 11973 raft_consensus.cc:504] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.506179 11609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4ee02a7172444b9cd64c46f2c82f26" candidate_uuid: "28dcee2af0984fb580286b5f85f6ee6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" is_pre_election: true
I20260924 07:06:21.506280 11609 raft_consensus.cc:2454] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28dcee2af0984fb580286b5f85f6ee6e in term 0.
I20260924 07:06:21.506552 11973 leader_election.cc:288] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e [CANDIDATE]: Term 1 election: Requested vote from peers 0d49625a33fb4f1b818f190bf9b4ad9f (127.3.217.1:33363), 725254e86479418ab43dddee8a117821 (127.3.217.3:43785)
I20260924 07:06:21.507021 11609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4ee02a7172444b9cd64c46f2c82f26" candidate_uuid: "28dcee2af0984fb580286b5f85f6ee6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f"
I20260924 07:06:21.507125 11609 raft_consensus.cc:3037] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:21.508200 11609 raft_consensus.cc:2454] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28dcee2af0984fb580286b5f85f6ee6e in term 1.
I20260924 07:06:21.508473 11682 leader_election.cc:302] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d49625a33fb4f1b818f190bf9b4ad9f, 28dcee2af0984fb580286b5f85f6ee6e; no voters:
I20260924 07:06:21.509091 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4ee02a7172444b9cd64c46f2c82f26" candidate_uuid: "28dcee2af0984fb580286b5f85f6ee6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "725254e86479418ab43dddee8a117821"
I20260924 07:06:21.509310 11891 raft_consensus.cc:3037] T 7e4ee02a7172444b9cd64c46f2c82f26 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:21.510612 11962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.503465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:34247 (local address 127.3.217.2:43527)
0924 07:06:21.503609 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:21.503613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:21.503796 (+ 183us) server_negotiation.cc:407] Connection header received
0924 07:06:21.503969 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:21.503972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:21.504012 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:21.504052 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:21.504386 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.507239 (+ 2853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.508899 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.509652 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.509763 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.509827 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:21.510228 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 07:06:21.510314 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2068,"server-negotiator.queue_time_us":79}
I20260924 07:06:21.511227 11891 raft_consensus.cc:2454] T 7e4ee02a7172444b9cd64c46f2c82f26 P 725254e86479418ab43dddee8a117821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28dcee2af0984fb580286b5f85f6ee6e in term 1.
I20260924 07:06:21.511423 11995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.503398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:43527 (local address 127.3.217.1:34247)
0924 07:06:21.503730 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:21.503743 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:21.503859 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:21.504158 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:21.504161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:21.504176 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:21.504285 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.504290 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.507378 (+ 3088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.507381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.508564 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.508574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.509947 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.509950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.510094 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.510117 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:06:21.510884 (+ 767us) client_negotiation.cc:240] Negotiation successful
0924 07:06:21.511017 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":93,"threads_started":1}
I20260924 07:06:21.512101 11751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4ee02a7172444b9cd64c46f2c82f26" candidate_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dcee2af0984fb580286b5f85f6ee6e" is_pre_election: true
I20260924 07:06:21.512298 11751 raft_consensus.cc:2379] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d49625a33fb4f1b818f190bf9b4ad9f in current term 1: Already voted for candidate 28dcee2af0984fb580286b5f85f6ee6e in this term.
I20260924 07:06:21.512384 11973 raft_consensus.cc:2781] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:21.512902 11973 raft_consensus.cc:686] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e [term 1 LEADER]: Becoming Leader. State: Replica: 28dcee2af0984fb580286b5f85f6ee6e, State: Running, Role: LEADER
I20260924 07:06:21.513082 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.503851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:40843 (local address 127.3.217.3:43785)
0924 07:06:21.503942 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:21.503945 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:21.508039 (+ 4094us) server_negotiation.cc:407] Connection header received
0924 07:06:21.508205 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:21.508208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:21.508249 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:21.508308 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:21.509626 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.510352 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.511815 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.512434 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.512507 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.512564 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:21.512879 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 07:06:21.512960 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:06:21.513252 11973 consensus_queue.cc:237] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.513455 11996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.503797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:43785 (local address 127.3.217.1:40843)
0924 07:06:21.507937 (+ 4140us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:21.507953 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:21.508102 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:21.509381 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:21.509384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:21.509402 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:21.509512 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.509518 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.510483 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.510486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.511690 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.511701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.512653 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.512657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.512757 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.512779 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:06:21.513238 (+ 459us) client_negotiation.cc:240] Negotiation successful
0924 07:06:21.513319 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4057,"thread_start_us":104,"threads_started":1}
I20260924 07:06:21.513962 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4ee02a7172444b9cd64c46f2c82f26" candidate_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "725254e86479418ab43dddee8a117821" is_pre_election: true
I20260924 07:06:21.514176 11890 raft_consensus.cc:2379] T 7e4ee02a7172444b9cd64c46f2c82f26 P 725254e86479418ab43dddee8a117821 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d49625a33fb4f1b818f190bf9b4ad9f in current term 1: Already voted for candidate 28dcee2af0984fb580286b5f85f6ee6e in this term.
I20260924 07:06:21.515043 11542 leader_election.cc:302] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d49625a33fb4f1b818f190bf9b4ad9f; no voters: 28dcee2af0984fb580286b5f85f6ee6e, 725254e86479418ab43dddee8a117821
I20260924 07:06:21.515488 11974 raft_consensus.cc:2726] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 07:06:21.517359 11451 catalog_manager.cc:5495] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 28dcee2af0984fb580286b5f85f6ee6e (127.3.217.2). New cstate: current_term: 1 leader_uuid: "28dcee2af0984fb580286b5f85f6ee6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } health_report { overall_health: UNKNOWN } } }
W20260924 07:06:21.526477 11798 tablet.cc:2369] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:06:21.540135 11939 tablet.cc:2369] T 7e4ee02a7172444b9cd64c46f2c82f26 P 725254e86479418ab43dddee8a117821: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:06:21.545328 11658 tablet.cc:2369] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 07:06:21 dist-test-slave-2p7j krb5kdc[11408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233580, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.62@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM
I20260924 07:06:21.564759 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.554206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54410 (local address 127.3.217.62:43511)
0924 07:06:21.554520 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:21.554525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:21.554689 (+ 164us) server_negotiation.cc:407] Connection header received
0924 07:06:21.554814 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:21.554818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:21.554857 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:21.554923 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:21.555960 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.556715 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.557908 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.558432 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.558644 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.561082 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:21.561085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:21.561087 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:21.561107 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:21.562637 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:21.563086 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:21.563089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:21.563090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:21.563155 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:21.563568 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:21.563575 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:21.563577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:21.563948 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:21.564159 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:21.564537 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 07:06:21.564635 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
I20260924 07:06:21.564759 12004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.554043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:43511 (local address 127.0.0.1:54410)
0924 07:06:21.554538 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:21.554553 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:21.554711 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:21.554960 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:21.554962 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:21.555577 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:21.555711 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.555722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.556871 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.556875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.557612 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.557629 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.558664 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.558667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.558762 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.559446 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:21.559458 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:21.560948 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:21.562756 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:21.562765 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:21.562770 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:21.562990 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:21.563240 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:21.563243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:21.563245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:21.563319 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:21.564179 (+ 860us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:21.564185 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:21.564313 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 07:06:21.564413 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 07:06:21.564567 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":142,"threads_started":1}
I20260924 07:06:21.570151 11451 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:54410:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 07:06:21.570947 11451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:06:21.575714 11591 tablet_service.cc:1467] Processing CreateTablet for tablet c0df182b8a8e4c95915e20434b61b6e8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1788b41fba2d44449a1317951b2addd4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:06:21.575969 11871 tablet_service.cc:1467] Processing CreateTablet for tablet c0df182b8a8e4c95915e20434b61b6e8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1788b41fba2d44449a1317951b2addd4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:06:21.576130 11591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0df182b8a8e4c95915e20434b61b6e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:21.576305 11871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0df182b8a8e4c95915e20434b61b6e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:21.578413 11966 tablet_bootstrap.cc:492] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821: Bootstrap starting.
I20260924 07:06:21.579268 11731 tablet_service.cc:1467] Processing CreateTablet for tablet c0df182b8a8e4c95915e20434b61b6e8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1788b41fba2d44449a1317951b2addd4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:06:21.579620 11731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0df182b8a8e4c95915e20434b61b6e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:21.579968 11966 tablet_bootstrap.cc:654] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:21.581027 11966 tablet_bootstrap.cc:492] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821: No bootstrap required, opened a new log
I20260924 07:06:21.581104 11966 ts_tablet_manager.cc:1397] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:06:21.581485 11966 raft_consensus.cc:348] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.581581 11966 raft_consensus.cc:374] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:21.581616 11966 raft_consensus.cc:729] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 725254e86479418ab43dddee8a117821, State: Initialized, Role: FOLLOWER
I20260924 07:06:21.581732 11966 consensus_queue.cc:260] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.581980 11976 tablet_replica.cc:439] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } } }
I20260924 07:06:21.582094 11976 tablet_replica.cc:442] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:06:21.582389 11966 ts_tablet_manager.cc:1428] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:06:21.583732 11967 tablet_bootstrap.cc:492] T c0df182b8a8e4c95915e20434b61b6e8 P 0d49625a33fb4f1b818f190bf9b4ad9f: Bootstrap starting.
I20260924 07:06:21.584724 11967 tablet_bootstrap.cc:654] T c0df182b8a8e4c95915e20434b61b6e8 P 0d49625a33fb4f1b818f190bf9b4ad9f: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:21.585769 11968 tablet_bootstrap.cc:492] T c0df182b8a8e4c95915e20434b61b6e8 P 28dcee2af0984fb580286b5f85f6ee6e: Bootstrap starting.
I20260924 07:06:21.586251 11967 tablet_bootstrap.cc:492] T c0df182b8a8e4c95915e20434b61b6e8 P 0d49625a33fb4f1b818f190bf9b4ad9f: No bootstrap required, opened a new log
I20260924 07:06:21.586335 11967 ts_tablet_manager.cc:1397] T c0df182b8a8e4c95915e20434b61b6e8 P 0d49625a33fb4f1b818f190bf9b4ad9f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:06:21.586771 11968 tablet_bootstrap.cc:654] T c0df182b8a8e4c95915e20434b61b6e8 P 28dcee2af0984fb580286b5f85f6ee6e: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:21.586853 11967 raft_consensus.cc:348] T c0df182b8a8e4c95915e20434b61b6e8 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.587009 11967 raft_consensus.cc:374] T c0df182b8a8e4c95915e20434b61b6e8 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:21.587054 11967 raft_consensus.cc:729] T c0df182b8a8e4c95915e20434b61b6e8 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d49625a33fb4f1b818f190bf9b4ad9f, State: Initialized, Role: FOLLOWER
I20260924 07:06:21.587178 11967 consensus_queue.cc:260] T c0df182b8a8e4c95915e20434b61b6e8 P 0d49625a33fb4f1b818f190bf9b4ad9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.587527 11967 ts_tablet_manager.cc:1428] T c0df182b8a8e4c95915e20434b61b6e8 P 0d49625a33fb4f1b818f190bf9b4ad9f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:06:21.587495 11974 tablet_replica.cc:439] T c0df182b8a8e4c95915e20434b61b6e8 P 0d49625a33fb4f1b818f190bf9b4ad9f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } } }
I20260924 07:06:21.587643 11974 tablet_replica.cc:442] T c0df182b8a8e4c95915e20434b61b6e8 P 0d49625a33fb4f1b818f190bf9b4ad9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:06:21.589110 11968 tablet_bootstrap.cc:492] T c0df182b8a8e4c95915e20434b61b6e8 P 28dcee2af0984fb580286b5f85f6ee6e: No bootstrap required, opened a new log
I20260924 07:06:21.589205 11968 ts_tablet_manager.cc:1397] T c0df182b8a8e4c95915e20434b61b6e8 P 28dcee2af0984fb580286b5f85f6ee6e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260924 07:06:21.589671 11968 raft_consensus.cc:348] T c0df182b8a8e4c95915e20434b61b6e8 P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.589797 11968 raft_consensus.cc:374] T c0df182b8a8e4c95915e20434b61b6e8 P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:21.589839 11968 raft_consensus.cc:729] T c0df182b8a8e4c95915e20434b61b6e8 P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28dcee2af0984fb580286b5f85f6ee6e, State: Initialized, Role: FOLLOWER
I20260924 07:06:21.589982 11968 consensus_queue.cc:260] T c0df182b8a8e4c95915e20434b61b6e8 P 28dcee2af0984fb580286b5f85f6ee6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.590252 11968 ts_tablet_manager.cc:1428] T c0df182b8a8e4c95915e20434b61b6e8 P 28dcee2af0984fb580286b5f85f6ee6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:06:21.590219 11973 tablet_replica.cc:439] T c0df182b8a8e4c95915e20434b61b6e8 P 28dcee2af0984fb580286b5f85f6ee6e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } } }
I20260924 07:06:21.590337 11973 tablet_replica.cc:442] T c0df182b8a8e4c95915e20434b61b6e8 P 28dcee2af0984fb580286b5f85f6ee6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:06:21.591115 11976 raft_consensus.cc:482] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:06:21.591202 11976 raft_consensus.cc:504] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.592060 11976 leader_election.cc:288] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d49625a33fb4f1b818f190bf9b4ad9f (127.3.217.1:33363), 28dcee2af0984fb580286b5f85f6ee6e (127.3.217.2:43527)
I20260924 07:06:21.592669 11751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0df182b8a8e4c95915e20434b61b6e8" candidate_uuid: "725254e86479418ab43dddee8a117821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dcee2af0984fb580286b5f85f6ee6e" is_pre_election: true
I20260924 07:06:21.592821 11751 raft_consensus.cc:2454] T c0df182b8a8e4c95915e20434b61b6e8 P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 725254e86479418ab43dddee8a117821 in term 0.
I20260924 07:06:21.593160 11825 leader_election.cc:302] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28dcee2af0984fb580286b5f85f6ee6e, 725254e86479418ab43dddee8a117821; no voters:
I20260924 07:06:21.593382 11976 raft_consensus.cc:2781] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:06:21.593434 11976 raft_consensus.cc:482] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:06:21.593386 11609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0df182b8a8e4c95915e20434b61b6e8" candidate_uuid: "725254e86479418ab43dddee8a117821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" is_pre_election: true
I20260924 07:06:21.593467 11976 raft_consensus.cc:3037] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:21.593538 11609 raft_consensus.cc:2454] T c0df182b8a8e4c95915e20434b61b6e8 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 725254e86479418ab43dddee8a117821 in term 0.
I20260924 07:06:21.595119 11976 raft_consensus.cc:504] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.595460 11976 leader_election.cc:288] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821 [CANDIDATE]: Term 1 election: Requested vote from peers 0d49625a33fb4f1b818f190bf9b4ad9f (127.3.217.1:33363), 28dcee2af0984fb580286b5f85f6ee6e (127.3.217.2:43527)
I20260924 07:06:21.595793 11609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0df182b8a8e4c95915e20434b61b6e8" candidate_uuid: "725254e86479418ab43dddee8a117821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f"
I20260924 07:06:21.595919 11609 raft_consensus.cc:3037] T c0df182b8a8e4c95915e20434b61b6e8 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:21.595945 11751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0df182b8a8e4c95915e20434b61b6e8" candidate_uuid: "725254e86479418ab43dddee8a117821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dcee2af0984fb580286b5f85f6ee6e"
I20260924 07:06:21.596100 11751 raft_consensus.cc:3037] T c0df182b8a8e4c95915e20434b61b6e8 P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:21.596655 11609 raft_consensus.cc:2454] T c0df182b8a8e4c95915e20434b61b6e8 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 725254e86479418ab43dddee8a117821 in term 1.
I20260924 07:06:21.596953 11822 leader_election.cc:302] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d49625a33fb4f1b818f190bf9b4ad9f, 725254e86479418ab43dddee8a117821; no voters:
I20260924 07:06:21.597103 11976 raft_consensus.cc:2781] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:21.597174 11976 raft_consensus.cc:686] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821 [term 1 LEADER]: Becoming Leader. State: Replica: 725254e86479418ab43dddee8a117821, State: Running, Role: LEADER
I20260924 07:06:21.597280 11976 consensus_queue.cc:237] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.597777 11751 raft_consensus.cc:2454] T c0df182b8a8e4c95915e20434b61b6e8 P 28dcee2af0984fb580286b5f85f6ee6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 725254e86479418ab43dddee8a117821 in term 1.
I20260924 07:06:21.597934 11972 tablet_replica.cc:439] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821: TxnStatusTablet state changed. Reason: New leader 725254e86479418ab43dddee8a117821. Latest consensus state: current_term: 1 leader_uuid: "725254e86479418ab43dddee8a117821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } } }
I20260924 07:06:21.598035 11972 tablet_replica.cc:442] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:06:21.598704 12009 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 07:06:21.598840 12009 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 07:06:21.599962 11451 catalog_manager.cc:5495] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 725254e86479418ab43dddee8a117821 (127.3.217.3). New cstate: current_term: 1 leader_uuid: "725254e86479418ab43dddee8a117821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } health_report { overall_health: HEALTHY } } }
I20260924 07:06:21.616034 12004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.611681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:43785 (local address 127.0.0.1:54972)
0924 07:06:21.611917 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:21.611932 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:21.612045 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:21.612351 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:21.612354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:21.612364 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:06:21.612485 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.612493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.613506 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.613510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.614517 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.614528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.615115 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.615119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.615210 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.615263 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:06:21.615702 (+ 439us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:06:21.615718 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 07:06:21.615797 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 07:06:21.615883 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260924 07:06:21.616077 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.611776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54972 (local address 127.3.217.3:43785)
0924 07:06:21.611915 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:21.611919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:21.611977 (+ 58us) server_negotiation.cc:407] Connection header received
0924 07:06:21.612192 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:21.612194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:21.612234 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:21.612294 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:06:21.612731 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.613330 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.614706 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.614972 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.615080 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.615450 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:06:21.615600 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:06:21.615681 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:21.615870 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 07:06:21.615938 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:06:21.619005 11751 raft_consensus.cc:1261] T c0df182b8a8e4c95915e20434b61b6e8 P 28dcee2af0984fb580286b5f85f6ee6e [term 1 FOLLOWER]: Refusing update from remote peer 725254e86479418ab43dddee8a117821: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:06:21.618999 11609 raft_consensus.cc:1261] T c0df182b8a8e4c95915e20434b61b6e8 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 1 FOLLOWER]: Refusing update from remote peer 725254e86479418ab43dddee8a117821: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:06:21.619457 11972 consensus_queue.cc:1035] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:06:21.619661 11976 consensus_queue.cc:1035] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:06:21.620990 11974 tablet_replica.cc:439] T c0df182b8a8e4c95915e20434b61b6e8 P 0d49625a33fb4f1b818f190bf9b4ad9f: TxnStatusTablet state changed. Reason: New leader 725254e86479418ab43dddee8a117821. Latest consensus state: current_term: 1 leader_uuid: "725254e86479418ab43dddee8a117821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } } }
I20260924 07:06:21.621107 11974 tablet_replica.cc:442] T c0df182b8a8e4c95915e20434b61b6e8 P 0d49625a33fb4f1b818f190bf9b4ad9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:06:21.622138 11973 tablet_replica.cc:439] T c0df182b8a8e4c95915e20434b61b6e8 P 28dcee2af0984fb580286b5f85f6ee6e: TxnStatusTablet state changed. Reason: New leader 725254e86479418ab43dddee8a117821. Latest consensus state: current_term: 1 leader_uuid: "725254e86479418ab43dddee8a117821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } } }
I20260924 07:06:21.622257 11973 tablet_replica.cc:442] T c0df182b8a8e4c95915e20434b61b6e8 P 28dcee2af0984fb580286b5f85f6ee6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:06:21.622512 11976 tablet_replica.cc:439] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "725254e86479418ab43dddee8a117821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } } }
I20260924 07:06:21.622617 11976 tablet_replica.cc:442] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:06:21.622843 11979 mvcc.cc:205] Tried to move back new op lower bound from 7332796750306836480 to 7332796750223749120. Current Snapshot: MvccSnapshot[applied={T|T < 7332796750306836480 or (T in {7332796750306836480})}]
I20260924 07:06:21.624155 11974 tablet_replica.cc:439] T c0df182b8a8e4c95915e20434b61b6e8 P 0d49625a33fb4f1b818f190bf9b4ad9f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "725254e86479418ab43dddee8a117821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } } }
I20260924 07:06:21.624269 11974 tablet_replica.cc:442] T c0df182b8a8e4c95915e20434b61b6e8 P 0d49625a33fb4f1b818f190bf9b4ad9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:06:21.626519 11972 tablet_replica.cc:439] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "725254e86479418ab43dddee8a117821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } } }
I20260924 07:06:21.626617 11972 tablet_replica.cc:442] T c0df182b8a8e4c95915e20434b61b6e8 P 725254e86479418ab43dddee8a117821: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:06:21.627357 11973 tablet_replica.cc:439] T c0df182b8a8e4c95915e20434b61b6e8 P 28dcee2af0984fb580286b5f85f6ee6e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "725254e86479418ab43dddee8a117821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } } }
I20260924 07:06:21.627460 11973 tablet_replica.cc:442] T c0df182b8a8e4c95915e20434b61b6e8 P 28dcee2af0984fb580286b5f85f6ee6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:06:21.631798 11962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.627460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58992 (local address 127.3.217.2:43527)
0924 07:06:21.627640 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:21.627644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:21.627655 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:06:21.627719 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:21.627721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:21.627774 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:21.627868 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:06:21.628358 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.628924 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.630308 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.630524 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.630608 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.630874 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:06:21.631309 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:06:21.631418 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:21.631614 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 07:06:21.631670 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260924 07:06:21.642282 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.638346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:53059 (local address 127.3.217.3:43785)
0924 07:06:21.638516 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:21.638520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:21.638673 (+ 153us) server_negotiation.cc:407] Connection header received
0924 07:06:21.638771 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:21.638774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:21.638822 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:21.638895 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:06:21.639377 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.640153 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.641060 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.641411 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.641500 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.641793 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:06:21.641868 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:06:21.641945 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:21.642085 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 07:06:21.642138 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 07:06:21.642347 12016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.638222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:43785 (local address 127.3.217.2:53059)
0924 07:06:21.638582 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:21.638595 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:21.638682 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:21.638906 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:21.638908 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:21.638912 (+ 4us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:06:21.639284 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.639289 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.640316 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.640319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.640960 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.640968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.641537 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.641540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.641617 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.641659 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:06:21.641981 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:06:21.641987 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:06:21.642053 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 07:06:21.642117 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":93,"threads_started":1}
I20260924 07:06:21.648896 11609 raft_consensus.cc:1261] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 1 FOLLOWER]: Refusing update from remote peer 28dcee2af0984fb580286b5f85f6ee6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:06:21.648984 11891 raft_consensus.cc:1261] T 7e4ee02a7172444b9cd64c46f2c82f26 P 725254e86479418ab43dddee8a117821 [term 1 FOLLOWER]: Refusing update from remote peer 28dcee2af0984fb580286b5f85f6ee6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:06:21.649354 11973 consensus_queue.cc:1035] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:06:21.649533 11997 consensus_queue.cc:1035] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:06:21.674513 11962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.670756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:49417 (local address 127.3.217.2:43527)
0924 07:06:21.670876 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:21.670879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:21.670890 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:06:21.670923 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:21.670924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:21.670959 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:21.671018 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:06:21.671245 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.671872 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.673405 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.673632 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.673738 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.674060 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:06:21.674148 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:06:21.674211 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:21.674339 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 07:06:21.674387 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 07:06:21.674533 11944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.670409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:43527 (local address 127.3.217.3:49417)
0924 07:06:21.670564 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:21.670578 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:21.670649 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:21.671050 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:21.671053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:21.671060 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:06:21.671151 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.671156 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.672027 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.672033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.673257 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:21.673269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.673761 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.673765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:21.673872 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.673922 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:06:21.674239 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:06:21.674245 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:06:21.674314 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 07:06:21.674427 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 07:06:21.697640 11452 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:06:21.697808 11452 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:06:21.699843 11452 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040: Sending DeleteTablet for 3 replicas of tablet 7e4ee02a7172444b9cd64c46f2c82f26
I20260924 07:06:21.700487 11591 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e4ee02a7172444b9cd64c46f2c82f26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:21 UTC) from {username='oryx', principal='oryx/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:44384
I20260924 07:06:21.700565 11871 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e4ee02a7172444b9cd64c46f2c82f26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:21 UTC) from {username='oryx', principal='oryx/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:54942
I20260924 07:06:21.700656 11988 tablet_replica.cc:330] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f: stopping tablet replica
I20260924 07:06:21.700724 11990 tablet_replica.cc:330] T 7e4ee02a7172444b9cd64c46f2c82f26 P 725254e86479418ab43dddee8a117821: stopping tablet replica
I20260924 07:06:21.700740 11988 raft_consensus.cc:2229] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:06:21.700824 11990 raft_consensus.cc:2229] T 7e4ee02a7172444b9cd64c46f2c82f26 P 725254e86479418ab43dddee8a117821 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:06:21.700840 11988 raft_consensus.cc:2258] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:21.700913 11990 raft_consensus.cc:2258] T 7e4ee02a7172444b9cd64c46f2c82f26 P 725254e86479418ab43dddee8a117821 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:21.701566 11988 ts_tablet_manager.cc:1905] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:06:21.701782 11990 ts_tablet_manager.cc:1905] T 7e4ee02a7172444b9cd64c46f2c82f26 P 725254e86479418ab43dddee8a117821: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:06:21.703796 11731 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e4ee02a7172444b9cd64c46f2c82f26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:21 UTC) from {username='oryx', principal='oryx/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:58986
I20260924 07:06:21.703960 11989 tablet_replica.cc:330] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e: stopping tablet replica
I20260924 07:06:21.704077 11989 raft_consensus.cc:2229] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:06:21.704350 11989 raft_consensus.cc:2258] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:21.704906 11990 ts_tablet_manager.cc:1918] T 7e4ee02a7172444b9cd64c46f2c82f26 P 725254e86479418ab43dddee8a117821: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:06:21.704968 11990 log.cc:1196] T 7e4ee02a7172444b9cd64c46f2c82f26 P 725254e86479418ab43dddee8a117821: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/7e4ee02a7172444b9cd64c46f2c82f26
I20260924 07:06:21.704984 11989 ts_tablet_manager.cc:1905] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:06:21.705178 11990 ts_tablet_manager.cc:1939] T 7e4ee02a7172444b9cd64c46f2c82f26 P 725254e86479418ab43dddee8a117821: Deleting consensus metadata
I20260924 07:06:21.707396 11989 ts_tablet_manager.cc:1918] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:06:21.707469 11989 log.cc:1196] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/7e4ee02a7172444b9cd64c46f2c82f26
I20260924 07:06:21.707729 11989 ts_tablet_manager.cc:1939] T 7e4ee02a7172444b9cd64c46f2c82f26 P 28dcee2af0984fb580286b5f85f6ee6e: Deleting consensus metadata
I20260924 07:06:21.707999 11437 catalog_manager.cc:4841] TS 725254e86479418ab43dddee8a117821 (127.3.217.3:43785): tablet 7e4ee02a7172444b9cd64c46f2c82f26 (table test-table [id=b9daafacae7c4ff4aa816ccb2400d42a]) successfully deleted
I20260924 07:06:21.708768 11988 ts_tablet_manager.cc:1918] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:06:21.708851 11988 log.cc:1196] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-0/wal/wals/7e4ee02a7172444b9cd64c46f2c82f26
I20260924 07:06:21.709074 11988 ts_tablet_manager.cc:1939] T 7e4ee02a7172444b9cd64c46f2c82f26 P 0d49625a33fb4f1b818f190bf9b4ad9f: Deleting consensus metadata
I20260924 07:06:21.709704 11440 catalog_manager.cc:4841] TS 28dcee2af0984fb580286b5f85f6ee6e (127.3.217.2:43527): tablet 7e4ee02a7172444b9cd64c46f2c82f26 (table test-table [id=b9daafacae7c4ff4aa816ccb2400d42a]) successfully deleted
I20260924 07:06:21.710384 11437 catalog_manager.cc:4841] TS 0d49625a33fb4f1b818f190bf9b4ad9f (127.3.217.1:33363): tablet 7e4ee02a7172444b9cd64c46f2c82f26 (table test-table [id=b9daafacae7c4ff4aa816ccb2400d42a]) successfully deleted
Sep 24 07:06:21 dist-test-slave-2p7j krb5kdc[11408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 07:06:21 dist-test-slave-2p7j krb5kdc[11408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:21 dist-test-slave-2p7j krb5kdc[11408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 07:06:21.736719 11503 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:21.729531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54422 (local address 127.3.217.62:43511)
0924 07:06:21.729646 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:21.729651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:21.729666 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:06:21.729711 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:21.729715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:21.729763 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:21.729846 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:21.730662 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.731470 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.732806 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.733159 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.733251 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.736600 (+ 3349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54422: BlockingRecv error: recv got EOF from 127.0.0.1:54422 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 07:06:21 dist-test-slave-2p7j krb5kdc[11408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM
I20260924 07:06:21.748353 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.738503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.3.217.62:43511)
0924 07:06:21.738595 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:21.738610 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:21.738977 (+ 367us) server_negotiation.cc:407] Connection header received
0924 07:06:21.739142 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:21.739146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:21.739191 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:21.739274 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:21.740166 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.740932 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.742019 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.742395 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.742493 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.745230 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:21.745234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:21.745237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:21.745261 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:21.746763 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:21.747190 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:21.747193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:21.747195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:21.747269 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:21.747538 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:21.747541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:21.747543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:21.747742 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:21.747864 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:21.748092 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 07:06:21.748197 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 07:06:21.751217 11452 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:06:21.751715 11452 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:06:21.756578 11871 tablet_service.cc:1467] Processing CreateTablet for tablet 073758b5fd2d4dee8650292602a1125a (DEFAULT_TABLE table=test-table [id=88c8c7800fa345f8bc79e03615c3f63b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:21.756415 11591 tablet_service.cc:1467] Processing CreateTablet for tablet 073758b5fd2d4dee8650292602a1125a (DEFAULT_TABLE table=test-table [id=88c8c7800fa345f8bc79e03615c3f63b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:21.756897 11591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 073758b5fd2d4dee8650292602a1125a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:21.757306 11871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 073758b5fd2d4dee8650292602a1125a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:21.760977 11967 tablet_bootstrap.cc:492] T 073758b5fd2d4dee8650292602a1125a P 0d49625a33fb4f1b818f190bf9b4ad9f: Bootstrap starting.
I20260924 07:06:21.761945 11967 tablet_bootstrap.cc:654] T 073758b5fd2d4dee8650292602a1125a P 0d49625a33fb4f1b818f190bf9b4ad9f: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:21.761992 11731 tablet_service.cc:1467] Processing CreateTablet for tablet 073758b5fd2d4dee8650292602a1125a (DEFAULT_TABLE table=test-table [id=88c8c7800fa345f8bc79e03615c3f63b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:21.762372 11731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 073758b5fd2d4dee8650292602a1125a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:21.765200 11968 tablet_bootstrap.cc:492] T 073758b5fd2d4dee8650292602a1125a P 28dcee2af0984fb580286b5f85f6ee6e: Bootstrap starting.
I20260924 07:06:21.766165 11968 tablet_bootstrap.cc:654] T 073758b5fd2d4dee8650292602a1125a P 28dcee2af0984fb580286b5f85f6ee6e: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:21.767159 11966 tablet_bootstrap.cc:492] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821: Bootstrap starting.
I20260924 07:06:21.767879 11967 tablet_bootstrap.cc:492] T 073758b5fd2d4dee8650292602a1125a P 0d49625a33fb4f1b818f190bf9b4ad9f: No bootstrap required, opened a new log
I20260924 07:06:21.767992 11966 tablet_bootstrap.cc:654] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:21.768016 11967 ts_tablet_manager.cc:1397] T 073758b5fd2d4dee8650292602a1125a P 0d49625a33fb4f1b818f190bf9b4ad9f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260924 07:06:21.768805 11967 raft_consensus.cc:348] T 073758b5fd2d4dee8650292602a1125a P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.768934 11967 raft_consensus.cc:374] T 073758b5fd2d4dee8650292602a1125a P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:21.768962 11967 raft_consensus.cc:729] T 073758b5fd2d4dee8650292602a1125a P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d49625a33fb4f1b818f190bf9b4ad9f, State: Initialized, Role: FOLLOWER
I20260924 07:06:21.769088 11967 consensus_queue.cc:260] T 073758b5fd2d4dee8650292602a1125a P 0d49625a33fb4f1b818f190bf9b4ad9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.769335 11967 ts_tablet_manager.cc:1428] T 073758b5fd2d4dee8650292602a1125a P 0d49625a33fb4f1b818f190bf9b4ad9f: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260924 07:06:21.771018 11966 tablet_bootstrap.cc:492] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821: No bootstrap required, opened a new log
I20260924 07:06:21.771201 11966 ts_tablet_manager.cc:1397] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 07:06:21.771768 11966 raft_consensus.cc:348] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.771919 11966 raft_consensus.cc:374] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:21.771953 11966 raft_consensus.cc:729] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 725254e86479418ab43dddee8a117821, State: Initialized, Role: FOLLOWER
I20260924 07:06:21.772112 11966 consensus_queue.cc:260] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.772428 11966 ts_tablet_manager.cc:1428] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:06:21.774298 11968 tablet_bootstrap.cc:492] T 073758b5fd2d4dee8650292602a1125a P 28dcee2af0984fb580286b5f85f6ee6e: No bootstrap required, opened a new log
I20260924 07:06:21.774418 11968 ts_tablet_manager.cc:1397] T 073758b5fd2d4dee8650292602a1125a P 28dcee2af0984fb580286b5f85f6ee6e: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20260924 07:06:21.774936 11968 raft_consensus.cc:348] T 073758b5fd2d4dee8650292602a1125a P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.775082 11968 raft_consensus.cc:374] T 073758b5fd2d4dee8650292602a1125a P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:21.775115 11968 raft_consensus.cc:729] T 073758b5fd2d4dee8650292602a1125a P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28dcee2af0984fb580286b5f85f6ee6e, State: Initialized, Role: FOLLOWER
I20260924 07:06:21.775246 11968 consensus_queue.cc:260] T 073758b5fd2d4dee8650292602a1125a P 28dcee2af0984fb580286b5f85f6ee6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.775568 11968 ts_tablet_manager.cc:1428] T 073758b5fd2d4dee8650292602a1125a P 28dcee2af0984fb580286b5f85f6ee6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:06:21.831254 11972 raft_consensus.cc:482] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:06:21.831414 11972 raft_consensus.cc:504] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.831861 11972 leader_election.cc:288] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d49625a33fb4f1b818f190bf9b4ad9f (127.3.217.1:33363), 28dcee2af0984fb580286b5f85f6ee6e (127.3.217.2:43527)
I20260924 07:06:21.832237 11751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073758b5fd2d4dee8650292602a1125a" candidate_uuid: "725254e86479418ab43dddee8a117821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dcee2af0984fb580286b5f85f6ee6e" is_pre_election: true
I20260924 07:06:21.832273 11609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073758b5fd2d4dee8650292602a1125a" candidate_uuid: "725254e86479418ab43dddee8a117821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" is_pre_election: true
I20260924 07:06:21.832401 11751 raft_consensus.cc:2454] T 073758b5fd2d4dee8650292602a1125a P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 725254e86479418ab43dddee8a117821 in term 0.
I20260924 07:06:21.832424 11609 raft_consensus.cc:2454] T 073758b5fd2d4dee8650292602a1125a P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 725254e86479418ab43dddee8a117821 in term 0.
I20260924 07:06:21.832682 11825 leader_election.cc:302] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28dcee2af0984fb580286b5f85f6ee6e, 725254e86479418ab43dddee8a117821; no voters:
I20260924 07:06:21.832849 11972 raft_consensus.cc:2781] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:06:21.832911 11972 raft_consensus.cc:482] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:06:21.832949 11972 raft_consensus.cc:3037] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:21.834003 11972 raft_consensus.cc:504] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.834275 11972 leader_election.cc:288] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821 [CANDIDATE]: Term 1 election: Requested vote from peers 0d49625a33fb4f1b818f190bf9b4ad9f (127.3.217.1:33363), 28dcee2af0984fb580286b5f85f6ee6e (127.3.217.2:43527)
I20260924 07:06:21.834563 11609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073758b5fd2d4dee8650292602a1125a" candidate_uuid: "725254e86479418ab43dddee8a117821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f"
I20260924 07:06:21.834554 11751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073758b5fd2d4dee8650292602a1125a" candidate_uuid: "725254e86479418ab43dddee8a117821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dcee2af0984fb580286b5f85f6ee6e"
I20260924 07:06:21.834648 11751 raft_consensus.cc:3037] T 073758b5fd2d4dee8650292602a1125a P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:21.834647 11609 raft_consensus.cc:3037] T 073758b5fd2d4dee8650292602a1125a P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:21.835675 11609 raft_consensus.cc:2454] T 073758b5fd2d4dee8650292602a1125a P 0d49625a33fb4f1b818f190bf9b4ad9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 725254e86479418ab43dddee8a117821 in term 1.
I20260924 07:06:21.835675 11751 raft_consensus.cc:2454] T 073758b5fd2d4dee8650292602a1125a P 28dcee2af0984fb580286b5f85f6ee6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 725254e86479418ab43dddee8a117821 in term 1.
I20260924 07:06:21.835927 11825 leader_election.cc:302] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28dcee2af0984fb580286b5f85f6ee6e, 725254e86479418ab43dddee8a117821; no voters:
I20260924 07:06:21.836118 11972 raft_consensus.cc:2781] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:21.836198 11972 raft_consensus.cc:686] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821 [term 1 LEADER]: Becoming Leader. State: Replica: 725254e86479418ab43dddee8a117821, State: Running, Role: LEADER
I20260924 07:06:21.836325 11972 consensus_queue.cc:237] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.838013 11451 catalog_manager.cc:5495] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 725254e86479418ab43dddee8a117821 (127.3.217.3). New cstate: current_term: 1 leader_uuid: "725254e86479418ab43dddee8a117821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } health_report { overall_health: HEALTHY } } }
I20260924 07:06:21.848790 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:21.845909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54974 (local address 127.3.217.3:43785)
0924 07:06:21.846043 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:21.846048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:21.846060 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:06:21.846173 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:21.846176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:21.846222 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:21.846280 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:06:21.846528 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.847020 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.847880 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:21.848092 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:21.848174 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:21.848395 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:06:21.848459 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:06:21.848515 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:21.848618 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 07:06:21.848664 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 07:06:21.850675 11609 raft_consensus.cc:1261] T 073758b5fd2d4dee8650292602a1125a P 0d49625a33fb4f1b818f190bf9b4ad9f [term 1 FOLLOWER]: Refusing update from remote peer 725254e86479418ab43dddee8a117821: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:06:21.850675 11751 raft_consensus.cc:1261] T 073758b5fd2d4dee8650292602a1125a P 28dcee2af0984fb580286b5f85f6ee6e [term 1 FOLLOWER]: Refusing update from remote peer 725254e86479418ab43dddee8a117821: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:06:21.851116 11972 consensus_queue.cc:1035] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:06:21.851428 11976 consensus_queue.cc:1035] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:06:21.860132 11453 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:06:21.860280 11453 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:06:21.862696 11453 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040: Sending DeleteTablet for 3 replicas of tablet 073758b5fd2d4dee8650292602a1125a
I20260924 07:06:21.863384 11591 tablet_service.cc:1514] Processing DeleteTablet for tablet 073758b5fd2d4dee8650292602a1125a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:21 UTC) from {username='oryx', principal='oryx/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:44384
I20260924 07:06:21.863605 11988 tablet_replica.cc:330] T 073758b5fd2d4dee8650292602a1125a P 0d49625a33fb4f1b818f190bf9b4ad9f: stopping tablet replica
I20260924 07:06:21.863622 11871 tablet_service.cc:1514] Processing DeleteTablet for tablet 073758b5fd2d4dee8650292602a1125a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:21 UTC) from {username='oryx', principal='oryx/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:54942
I20260924 07:06:21.863721 11988 raft_consensus.cc:2229] T 073758b5fd2d4dee8650292602a1125a P 0d49625a33fb4f1b818f190bf9b4ad9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:06:21.863780 11990 tablet_replica.cc:330] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821: stopping tablet replica
I20260924 07:06:21.863873 11988 raft_consensus.cc:2258] T 073758b5fd2d4dee8650292602a1125a P 0d49625a33fb4f1b818f190bf9b4ad9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:21.863899 11990 raft_consensus.cc:2229] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:06:21.864108 11990 raft_consensus.cc:2258] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:21.864560 11988 ts_tablet_manager.cc:1905] T 073758b5fd2d4dee8650292602a1125a P 0d49625a33fb4f1b818f190bf9b4ad9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:06:21.864670 11731 tablet_service.cc:1514] Processing DeleteTablet for tablet 073758b5fd2d4dee8650292602a1125a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:21 UTC) from {username='oryx', principal='oryx/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:58986
I20260924 07:06:21.864916 11990 ts_tablet_manager.cc:1905] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:06:21.866576 11988 ts_tablet_manager.cc:1918] T 073758b5fd2d4dee8650292602a1125a P 0d49625a33fb4f1b818f190bf9b4ad9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:06:21.866641 11988 log.cc:1196] T 073758b5fd2d4dee8650292602a1125a P 0d49625a33fb4f1b818f190bf9b4ad9f: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-0/wal/wals/073758b5fd2d4dee8650292602a1125a
I20260924 07:06:21.866873 11988 ts_tablet_manager.cc:1939] T 073758b5fd2d4dee8650292602a1125a P 0d49625a33fb4f1b818f190bf9b4ad9f: Deleting consensus metadata
I20260924 07:06:21.867131 11990 ts_tablet_manager.cc:1918] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:06:21.867187 11990 log.cc:1196] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/073758b5fd2d4dee8650292602a1125a
I20260924 07:06:21.867341 11989 tablet_replica.cc:330] T 073758b5fd2d4dee8650292602a1125a P 28dcee2af0984fb580286b5f85f6ee6e: stopping tablet replica
I20260924 07:06:21.867384 11990 ts_tablet_manager.cc:1939] T 073758b5fd2d4dee8650292602a1125a P 725254e86479418ab43dddee8a117821: Deleting consensus metadata
I20260924 07:06:21.867452 11989 raft_consensus.cc:2229] T 073758b5fd2d4dee8650292602a1125a P 28dcee2af0984fb580286b5f85f6ee6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:06:21.867570 11437 catalog_manager.cc:4841] TS 0d49625a33fb4f1b818f190bf9b4ad9f (127.3.217.1:33363): tablet 073758b5fd2d4dee8650292602a1125a (table test-table [id=88c8c7800fa345f8bc79e03615c3f63b]) successfully deleted
I20260924 07:06:21.867748 11989 raft_consensus.cc:2258] T 073758b5fd2d4dee8650292602a1125a P 28dcee2af0984fb580286b5f85f6ee6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:21.868188 11437 catalog_manager.cc:4841] TS 725254e86479418ab43dddee8a117821 (127.3.217.3:43785): tablet 073758b5fd2d4dee8650292602a1125a (table test-table [id=88c8c7800fa345f8bc79e03615c3f63b]) successfully deleted
I20260924 07:06:21.868510 11989 ts_tablet_manager.cc:1905] T 073758b5fd2d4dee8650292602a1125a P 28dcee2af0984fb580286b5f85f6ee6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:06:21.868683 11452 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:06:21.869043 11452 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:06:21.870687 11989 ts_tablet_manager.cc:1918] T 073758b5fd2d4dee8650292602a1125a P 28dcee2af0984fb580286b5f85f6ee6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:06:21.870757 11989 log.cc:1196] T 073758b5fd2d4dee8650292602a1125a P 28dcee2af0984fb580286b5f85f6ee6e: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/073758b5fd2d4dee8650292602a1125a
I20260924 07:06:21.871017 11989 ts_tablet_manager.cc:1939] T 073758b5fd2d4dee8650292602a1125a P 28dcee2af0984fb580286b5f85f6ee6e: Deleting consensus metadata
I20260924 07:06:21.871845 11440 catalog_manager.cc:4841] TS 28dcee2af0984fb580286b5f85f6ee6e (127.3.217.2:43527): tablet 073758b5fd2d4dee8650292602a1125a (table test-table [id=88c8c7800fa345f8bc79e03615c3f63b]) successfully deleted
I20260924 07:06:21.873528 11591 tablet_service.cc:1467] Processing CreateTablet for tablet 0c77761c652740ff97c52c14d9413ad5 (DEFAULT_TABLE table=test-table [id=3504ebb97ba44557bdf45d18d9a2bda7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:21.873682 11731 tablet_service.cc:1467] Processing CreateTablet for tablet 0c77761c652740ff97c52c14d9413ad5 (DEFAULT_TABLE table=test-table [id=3504ebb97ba44557bdf45d18d9a2bda7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:21.873775 11591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c77761c652740ff97c52c14d9413ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:21.873903 11731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c77761c652740ff97c52c14d9413ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:21.875478 11967 tablet_bootstrap.cc:492] T 0c77761c652740ff97c52c14d9413ad5 P 0d49625a33fb4f1b818f190bf9b4ad9f: Bootstrap starting.
I20260924 07:06:21.875957 11968 tablet_bootstrap.cc:492] T 0c77761c652740ff97c52c14d9413ad5 P 28dcee2af0984fb580286b5f85f6ee6e: Bootstrap starting.
I20260924 07:06:21.876248 11967 tablet_bootstrap.cc:654] T 0c77761c652740ff97c52c14d9413ad5 P 0d49625a33fb4f1b818f190bf9b4ad9f: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:21.876763 11968 tablet_bootstrap.cc:654] T 0c77761c652740ff97c52c14d9413ad5 P 28dcee2af0984fb580286b5f85f6ee6e: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:21.877194 11967 tablet_bootstrap.cc:492] T 0c77761c652740ff97c52c14d9413ad5 P 0d49625a33fb4f1b818f190bf9b4ad9f: No bootstrap required, opened a new log
I20260924 07:06:21.877270 11967 ts_tablet_manager.cc:1397] T 0c77761c652740ff97c52c14d9413ad5 P 0d49625a33fb4f1b818f190bf9b4ad9f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 07:06:21.877620 11967 raft_consensus.cc:348] T 0c77761c652740ff97c52c14d9413ad5 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.877731 11967 raft_consensus.cc:374] T 0c77761c652740ff97c52c14d9413ad5 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:21.877768 11967 raft_consensus.cc:729] T 0c77761c652740ff97c52c14d9413ad5 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d49625a33fb4f1b818f190bf9b4ad9f, State: Initialized, Role: FOLLOWER
I20260924 07:06:21.877835 11871 tablet_service.cc:1467] Processing CreateTablet for tablet 0c77761c652740ff97c52c14d9413ad5 (DEFAULT_TABLE table=test-table [id=3504ebb97ba44557bdf45d18d9a2bda7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:21.877884 11967 consensus_queue.cc:260] T 0c77761c652740ff97c52c14d9413ad5 P 0d49625a33fb4f1b818f190bf9b4ad9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.878074 11871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c77761c652740ff97c52c14d9413ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:21.878132 11967 ts_tablet_manager.cc:1428] T 0c77761c652740ff97c52c14d9413ad5 P 0d49625a33fb4f1b818f190bf9b4ad9f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 07:06:21.880081 11968 tablet_bootstrap.cc:492] T 0c77761c652740ff97c52c14d9413ad5 P 28dcee2af0984fb580286b5f85f6ee6e: No bootstrap required, opened a new log
I20260924 07:06:21.880169 11968 ts_tablet_manager.cc:1397] T 0c77761c652740ff97c52c14d9413ad5 P 28dcee2af0984fb580286b5f85f6ee6e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 07:06:21.880779 11968 raft_consensus.cc:348] T 0c77761c652740ff97c52c14d9413ad5 P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.880896 11968 raft_consensus.cc:374] T 0c77761c652740ff97c52c14d9413ad5 P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:21.880934 11968 raft_consensus.cc:729] T 0c77761c652740ff97c52c14d9413ad5 P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28dcee2af0984fb580286b5f85f6ee6e, State: Initialized, Role: FOLLOWER
I20260924 07:06:21.881045 11968 consensus_queue.cc:260] T 0c77761c652740ff97c52c14d9413ad5 P 28dcee2af0984fb580286b5f85f6ee6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.881301 11968 ts_tablet_manager.cc:1428] T 0c77761c652740ff97c52c14d9413ad5 P 28dcee2af0984fb580286b5f85f6ee6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:06:21.882256 11966 tablet_bootstrap.cc:492] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821: Bootstrap starting.
I20260924 07:06:21.882921 11966 tablet_bootstrap.cc:654] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:21.883965 11966 tablet_bootstrap.cc:492] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821: No bootstrap required, opened a new log
I20260924 07:06:21.884052 11966 ts_tablet_manager.cc:1397] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:06:21.884438 11966 raft_consensus.cc:348] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.884548 11966 raft_consensus.cc:374] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:21.884580 11966 raft_consensus.cc:729] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 725254e86479418ab43dddee8a117821, State: Initialized, Role: FOLLOWER
I20260924 07:06:21.884687 11966 consensus_queue.cc:260] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:21.884922 11966 ts_tablet_manager.cc:1428] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:06:22.122406 11972 raft_consensus.cc:482] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:06:22.122543 11972 raft_consensus.cc:504] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:22.123052 11972 leader_election.cc:288] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d49625a33fb4f1b818f190bf9b4ad9f (127.3.217.1:33363), 28dcee2af0984fb580286b5f85f6ee6e (127.3.217.2:43527)
I20260924 07:06:22.123602 11609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c77761c652740ff97c52c14d9413ad5" candidate_uuid: "725254e86479418ab43dddee8a117821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" is_pre_election: true
I20260924 07:06:22.123601 11751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c77761c652740ff97c52c14d9413ad5" candidate_uuid: "725254e86479418ab43dddee8a117821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dcee2af0984fb580286b5f85f6ee6e" is_pre_election: true
I20260924 07:06:22.123752 11751 raft_consensus.cc:2454] T 0c77761c652740ff97c52c14d9413ad5 P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 725254e86479418ab43dddee8a117821 in term 0.
I20260924 07:06:22.123752 11609 raft_consensus.cc:2454] T 0c77761c652740ff97c52c14d9413ad5 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 725254e86479418ab43dddee8a117821 in term 0.
I20260924 07:06:22.124048 11825 leader_election.cc:302] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28dcee2af0984fb580286b5f85f6ee6e, 725254e86479418ab43dddee8a117821; no voters:
I20260924 07:06:22.124214 11972 raft_consensus.cc:2781] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:06:22.124260 11972 raft_consensus.cc:482] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:06:22.124298 11972 raft_consensus.cc:3037] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:22.125204 11972 raft_consensus.cc:504] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:22.125502 11972 leader_election.cc:288] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821 [CANDIDATE]: Term 1 election: Requested vote from peers 0d49625a33fb4f1b818f190bf9b4ad9f (127.3.217.1:33363), 28dcee2af0984fb580286b5f85f6ee6e (127.3.217.2:43527)
I20260924 07:06:22.125813 11751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c77761c652740ff97c52c14d9413ad5" candidate_uuid: "725254e86479418ab43dddee8a117821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dcee2af0984fb580286b5f85f6ee6e"
I20260924 07:06:22.125912 11751 raft_consensus.cc:3037] T 0c77761c652740ff97c52c14d9413ad5 P 28dcee2af0984fb580286b5f85f6ee6e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:22.126078 11609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c77761c652740ff97c52c14d9413ad5" candidate_uuid: "725254e86479418ab43dddee8a117821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f"
I20260924 07:06:22.126267 11609 raft_consensus.cc:3037] T 0c77761c652740ff97c52c14d9413ad5 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:22.126748 11751 raft_consensus.cc:2454] T 0c77761c652740ff97c52c14d9413ad5 P 28dcee2af0984fb580286b5f85f6ee6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 725254e86479418ab43dddee8a117821 in term 1.
I20260924 07:06:22.127046 11825 leader_election.cc:302] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28dcee2af0984fb580286b5f85f6ee6e, 725254e86479418ab43dddee8a117821; no voters:
I20260924 07:06:22.127209 11972 raft_consensus.cc:2781] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:22.127275 11972 raft_consensus.cc:686] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821 [term 1 LEADER]: Becoming Leader. State: Replica: 725254e86479418ab43dddee8a117821, State: Running, Role: LEADER
I20260924 07:06:22.127406 11972 consensus_queue.cc:237] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } }
I20260924 07:06:22.127661 11609 raft_consensus.cc:2454] T 0c77761c652740ff97c52c14d9413ad5 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 725254e86479418ab43dddee8a117821 in term 1.
I20260924 07:06:22.129249 11453 catalog_manager.cc:5495] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 725254e86479418ab43dddee8a117821 (127.3.217.3). New cstate: current_term: 1 leader_uuid: "725254e86479418ab43dddee8a117821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "725254e86479418ab43dddee8a117821" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 43785 } health_report { overall_health: HEALTHY } } }
I20260924 07:06:22.195278 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:22.192091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:41809 (local address 127.3.217.3:43785)
0924 07:06:22.192184 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:22.192187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:22.192195 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:06:22.192221 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:22.192223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:22.192253 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:22.192302 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:06:22.192559 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:22.193069 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:22.194007 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:22.194299 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:22.194425 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:22.194765 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:06:22.194867 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:06:22.194932 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:22.195125 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 07:06:22.195173 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:06:22.195286 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:22.191701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:43785 (local address 127.3.217.3:41809)
0924 07:06:22.191960 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:22.191973 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:22.192049 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:22.192346 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:22.192349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:22.192362 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:06:22.192458 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:22.192462 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:22.193186 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:22.193189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:22.193811 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:22.193819 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:22.194487 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:22.194491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:22.194573 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:22.194603 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:06:22.195009 (+ 406us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:06:22.195017 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 07:06:22.195085 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 07:06:22.195157 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
I20260924 07:06:22.200513 11609 raft_consensus.cc:1261] T 0c77761c652740ff97c52c14d9413ad5 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 1 FOLLOWER]: Refusing update from remote peer 725254e86479418ab43dddee8a117821: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:06:22.200490 11751 raft_consensus.cc:1261] T 0c77761c652740ff97c52c14d9413ad5 P 28dcee2af0984fb580286b5f85f6ee6e [term 1 FOLLOWER]: Refusing update from remote peer 725254e86479418ab43dddee8a117821: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:06:22.200842 11972 consensus_queue.cc:1035] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d49625a33fb4f1b818f190bf9b4ad9f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:06:22.201150 11972 consensus_queue.cc:1035] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28dcee2af0984fb580286b5f85f6ee6e" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:06:22.237898 11452 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:06:22.238085 11452 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:06:22.240329 11452 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a8fc551150745cca69f43e7273ce040: Sending DeleteTablet for 3 replicas of tablet 0c77761c652740ff97c52c14d9413ad5
I20260924 07:06:22.240882 11731 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c77761c652740ff97c52c14d9413ad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:22 UTC) from {username='oryx', principal='oryx/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:58986
I20260924 07:06:22.240906 11591 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c77761c652740ff97c52c14d9413ad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:22 UTC) from {username='oryx', principal='oryx/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:44384
I20260924 07:06:22.241067 11988 tablet_replica.cc:330] T 0c77761c652740ff97c52c14d9413ad5 P 0d49625a33fb4f1b818f190bf9b4ad9f: stopping tablet replica
I20260924 07:06:22.241067 11989 tablet_replica.cc:330] T 0c77761c652740ff97c52c14d9413ad5 P 28dcee2af0984fb580286b5f85f6ee6e: stopping tablet replica
I20260924 07:06:22.241178 11988 raft_consensus.cc:2229] T 0c77761c652740ff97c52c14d9413ad5 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:06:22.241179 11989 raft_consensus.cc:2229] T 0c77761c652740ff97c52c14d9413ad5 P 28dcee2af0984fb580286b5f85f6ee6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:06:22.241282 11989 raft_consensus.cc:2258] T 0c77761c652740ff97c52c14d9413ad5 P 28dcee2af0984fb580286b5f85f6ee6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:22.241289 11988 raft_consensus.cc:2258] T 0c77761c652740ff97c52c14d9413ad5 P 0d49625a33fb4f1b818f190bf9b4ad9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:22.241452 11869 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c77761c652740ff97c52c14d9413ad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:22 UTC) from {username='oryx', principal='oryx/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:54942
I20260924 07:06:22.241585 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 11526
I20260924 07:06:22.241621 11990 tablet_replica.cc:330] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821: stopping tablet replica
I20260924 07:06:22.241721 11990 raft_consensus.cc:2229] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:06:22.241884 11990 raft_consensus.cc:2258] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:22.242833 11990 ts_tablet_manager.cc:1905] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:06:22.243113 11989 ts_tablet_manager.cc:1905] T 0c77761c652740ff97c52c14d9413ad5 P 28dcee2af0984fb580286b5f85f6ee6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:06:22.246289 11989 ts_tablet_manager.cc:1918] T 0c77761c652740ff97c52c14d9413ad5 P 28dcee2af0984fb580286b5f85f6ee6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:06:22.246272 11990 ts_tablet_manager.cc:1918] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:06:22.246393 11989 log.cc:1196] T 0c77761c652740ff97c52c14d9413ad5 P 28dcee2af0984fb580286b5f85f6ee6e: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/0c77761c652740ff97c52c14d9413ad5
I20260924 07:06:22.246397 11990 log.cc:1196] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/0c77761c652740ff97c52c14d9413ad5
I20260924 07:06:22.246681 11989 ts_tablet_manager.cc:1939] T 0c77761c652740ff97c52c14d9413ad5 P 28dcee2af0984fb580286b5f85f6ee6e: Deleting consensus metadata
I20260924 07:06:22.246707 11990 ts_tablet_manager.cc:1939] T 0c77761c652740ff97c52c14d9413ad5 P 725254e86479418ab43dddee8a117821: Deleting consensus metadata
I20260924 07:06:22.247601 11440 catalog_manager.cc:4841] TS 28dcee2af0984fb580286b5f85f6ee6e (127.3.217.2:43527): tablet 0c77761c652740ff97c52c14d9413ad5 (table test-table [id=3504ebb97ba44557bdf45d18d9a2bda7]) successfully deleted
I20260924 07:06:22.247749 11437 catalog_manager.cc:4841] TS 725254e86479418ab43dddee8a117821 (127.3.217.3:43785): tablet 0c77761c652740ff97c52c14d9413ad5 (table test-table [id=3504ebb97ba44557bdf45d18d9a2bda7]) successfully deleted
W20260924 07:06:22.249841 11437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.217.1:33363 (error 108)
I20260924 07:06:22.250026 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 11666
I20260924 07:06:22.251286 12054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:22.250931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:33363 (local address 127.0.0.1:44390)
0924 07:06:22.251131 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:22.251196 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.1:33363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":55,"threads_started":1}
W20260924 07:06:22.251514 11437 catalog_manager.cc:4584] TS 0d49625a33fb4f1b818f190bf9b4ad9f (127.3.217.1:33363): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0c77761c652740ff97c52c14d9413ad5: Network error: Client connection negotiation failed: client connection to 127.3.217.1:33363: connect: Connection refused (error 111)
I20260924 07:06:22.258494 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 11806
I20260924 07:06:22.266489 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 11421
2026-09-24T07:06:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3083 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:06:22 dist-test-slave-2p7j krb5kdc[12057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:06:22 dist-test-slave-2p7j krb5kdc[12057](info): set up 2 sockets
Sep 24 07:06:22 dist-test-slave-2p7j krb5kdc[12057](info): commencing operation
krb5kdc: starting...
W20260924 07:06:23.273149 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:06:23 dist-test-slave-2p7j krb5kdc[12057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:06:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:06:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:06:23.404222 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:43265
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:43265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:23.516485 12070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:23.516732 12070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:23.516772 12070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:23.521349 12070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:06:23.521466 12070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:23.521486 12070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:23.521505 12070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:06:23.521521 12070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:06:23.529040 12070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:43265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:43265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:23.530045 12070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:23.530964 12070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:23.539374 12075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:23.539392 12076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:23.539382 12078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:23.539880 12070 server_base.cc:1034] running on GCE node
I20260924 07:06:23.540377 12070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:23.541456 12070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:23.542682 12070 hybrid_clock.cc:648] HybridClock initialized: now 1790233583542654 us; error 42 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T07:06:23Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1284 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:06:23 dist-test-slave-2p7j krb5kdc[12083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:06:23 dist-test-slave-2p7j krb5kdc[12083](info): set up 2 sockets
Sep 24 07:06:23 dist-test-slave-2p7j krb5kdc[12083](info): commencing operation
krb5kdc: starting...
W20260924 07:06:24.540485 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:06:24 dist-test-slave-2p7j krb5kdc[12083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:06:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:06:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:06:24.679973 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:42643
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:38419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:42643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:24.794049 12096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:24.794310 12096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:24.794376 12096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:24.798646 12096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:06:24.798708 12096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:24.798735 12096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:24.798756 12096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:06:24.798776 12096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:06:24.803910 12096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:38419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:42643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:42643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:24.804857 12096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:24.805802 12096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:24.814045 12104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:24.814169 12102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:24.814543 12101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:24.814935 12096 server_base.cc:1034] running on GCE node
I20260924 07:06:24.815726 12096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:24.817181 12096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:24.818410 12096 hybrid_clock.cc:648] HybridClock initialized: now 1790233584818342 us; error 93 us; skew 500 ppm
Sep 24 07:06:24 dist-test-slave-2p7j krb5kdc[12083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233584, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:24.827414 12096 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:06:24.828356 12096 webserver.cc:460] Webserver started at http://127.3.217.62:39733/ using document root <none> and password file <none>
I20260924 07:06:24.828895 12096 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:24.828966 12096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:24.829147 12096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:24.831084 12096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "f46c1e1bcc0b4d1cb01fabc3b2f4dbb9"
format_stamp: "Formatted at 2026-09-24 07:06:24 on dist-test-slave-2p7j"
server_key: "4dff7502992299aee5bcadd29baac952"
server_key_iv: "21394c35d2323f3493238ef35975822f"
server_key_version: "encryptionkey@0"
I20260924 07:06:24.831646 12096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "f46c1e1bcc0b4d1cb01fabc3b2f4dbb9"
format_stamp: "Formatted at 2026-09-24 07:06:24 on dist-test-slave-2p7j"
server_key: "4dff7502992299aee5bcadd29baac952"
server_key_iv: "21394c35d2323f3493238ef35975822f"
server_key_version: "encryptionkey@0"
I20260924 07:06:24.835158 12096 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:06:24.838006 12111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:24.839242 12096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:06:24.839344 12096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "f46c1e1bcc0b4d1cb01fabc3b2f4dbb9"
format_stamp: "Formatted at 2026-09-24 07:06:24 on dist-test-slave-2p7j"
server_key: "4dff7502992299aee5bcadd29baac952"
server_key_iv: "21394c35d2323f3493238ef35975822f"
server_key_version: "encryptionkey@0"
I20260924 07:06:24.839452 12096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:24.860929 12096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:24.863924 12096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:24.864111 12096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:24.871344 12096 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:42643
I20260924 07:06:24.871366 12163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:42643 every 8 connection(s)
I20260924 07:06:24.872355 12096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:06:24.874990 12164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:24.877732 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 12096
I20260924 07:06:24.877857 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:06:24.878139 3940 external_mini_cluster.cc:1428] Setting key 67d55f28b308b384cf9687f8b180e378
I20260924 07:06:24.880467 12164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9: Bootstrap starting.
I20260924 07:06:24.882663 12164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:24.883337 12164 log.cc:824] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:24.885197 12164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9: No bootstrap required, opened a new log
Sep 24 07:06:24 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:06:24.888466 12164 raft_consensus.cc:348] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46c1e1bcc0b4d1cb01fabc3b2f4dbb9" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42643 } }
I20260924 07:06:24.888692 12164 raft_consensus.cc:374] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:24.888784 12164 raft_consensus.cc:729] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46c1e1bcc0b4d1cb01fabc3b2f4dbb9, State: Initialized, Role: FOLLOWER
I20260924 07:06:24.889200 12164 consensus_queue.cc:260] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46c1e1bcc0b4d1cb01fabc3b2f4dbb9" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42643 } }
I20260924 07:06:24.889318 12164 raft_consensus.cc:388] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:06:24.889362 12164 raft_consensus.cc:482] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:06:24.889451 12164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:24.890188 12164 raft_consensus.cc:504] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46c1e1bcc0b4d1cb01fabc3b2f4dbb9" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42643 } }
I20260924 07:06:24.890507 12164 leader_election.cc:302] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f46c1e1bcc0b4d1cb01fabc3b2f4dbb9; no voters:
I20260924 07:06:24.890805 12164 leader_election.cc:288] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:06:24.891029 12169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:24.891474 12169 raft_consensus.cc:686] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [term 1 LEADER]: Becoming Leader. State: Replica: f46c1e1bcc0b4d1cb01fabc3b2f4dbb9, State: Running, Role: LEADER
I20260924 07:06:24.891980 12169 consensus_queue.cc:237] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46c1e1bcc0b4d1cb01fabc3b2f4dbb9" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42643 } }
I20260924 07:06:24.892690 12167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:24.879648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54442 (local address 127.3.217.62:42643)
0924 07:06:24.880056 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:24.880072 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:24.880105 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:06:24.880737 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:24.880754 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:24.881068 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:24.881327 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:24.882177 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:24.883188 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:24.884194 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:24.884644 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:24.884758 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:24.887401 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:24.887406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:24.887417 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:24.887442 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:24.889506 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:24.889938 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:24.889944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:24.889948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:24.890063 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:24.890351 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:24.890354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:24.890356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:24.890836 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:24.891012 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:24.891452 (+ 440us) server_negotiation.cc:299] Negotiation successful
0924 07:06:24.891779 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":122,"threads_started":1}
I20260924 07:06:24.893661 12171 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f46c1e1bcc0b4d1cb01fabc3b2f4dbb9. Latest consensus state: current_term: 1 leader_uuid: "f46c1e1bcc0b4d1cb01fabc3b2f4dbb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46c1e1bcc0b4d1cb01fabc3b2f4dbb9" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42643 } } }
I20260924 07:06:24.892078 12164 sys_catalog.cc:564] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:06:24.893885 12171 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:24.894755 12173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:06:24.895000 12170 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f46c1e1bcc0b4d1cb01fabc3b2f4dbb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46c1e1bcc0b4d1cb01fabc3b2f4dbb9" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42643 } } }
I20260924 07:06:24.895097 12170 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:24.897825 12173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:06:24.903978 12173 catalog_manager.cc:1347] Generated new cluster ID: a5e0e2d0c6cd4750b50b4ccb2bb42a33
I20260924 07:06:24.904050 12173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:06:24.920033 12173 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:06:24.920929 12173 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:06:24.929759 12173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9: Generated new TSK 0
I20260924 07:06:24.930368 12173 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
I20260924 07:06:24.967427 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42643
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:38419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:25.116689 12190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:25.116940 12190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:25.116981 12190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:25.121049 12190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:25.121114 12190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:25.121178 12190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:06:25.126348 12190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:38419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:25.127274 12190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:25.128185 12190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:25.137288 12198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:25.137270 12195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:25.137442 12196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:25.137997 12190 server_base.cc:1034] running on GCE node
I20260924 07:06:25.138557 12190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:25.139575 12190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:25.140790 12190 hybrid_clock.cc:648] HybridClock initialized: now 1790233585140762 us; error 40 us; skew 500 ppm
Sep 24 07:06:25 dist-test-slave-2p7j krb5kdc[12083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233585, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:25.144044 12190 init.cc:377] Logged in from keytab as kudu/127.3.217.1@KRBTEST.COM (short username kudu)
I20260924 07:06:25.144960 12190 webserver.cc:460] Webserver started at http://127.3.217.1:45931/ using document root <none> and password file <none>
I20260924 07:06:25.145849 12190 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:25.145964 12190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:25.146241 12190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:25.149430 12190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "686d5f5aa64544a59b2cc83c0bd483d1"
format_stamp: "Formatted at 2026-09-24 07:06:25 on dist-test-slave-2p7j"
server_key: "8836008b45ca7dfc00943d4d914ee957"
server_key_iv: "2e7281841ed56b7af6c214e89103657a"
server_key_version: "encryptionkey@0"
I20260924 07:06:25.150301 12190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "686d5f5aa64544a59b2cc83c0bd483d1"
format_stamp: "Formatted at 2026-09-24 07:06:25 on dist-test-slave-2p7j"
server_key: "8836008b45ca7dfc00943d4d914ee957"
server_key_iv: "2e7281841ed56b7af6c214e89103657a"
server_key_version: "encryptionkey@0"
I20260924 07:06:25.154340 12190 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 07:06:25.157547 12205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:25.158918 12190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:06:25.159029 12190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "686d5f5aa64544a59b2cc83c0bd483d1"
format_stamp: "Formatted at 2026-09-24 07:06:25 on dist-test-slave-2p7j"
server_key: "8836008b45ca7dfc00943d4d914ee957"
server_key_iv: "2e7281841ed56b7af6c214e89103657a"
server_key_version: "encryptionkey@0"
I20260924 07:06:25.159144 12190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:25.178265 12190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:25.181747 12190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:25.181937 12190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:25.182547 12190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:25.183323 12190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:25.183369 12190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:25.183420 12190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:25.183449 12190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:25.191937 12190 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:41703
I20260924 07:06:25.192011 12318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:41703 every 8 connection(s)
I20260924 07:06:25.192749 12190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:06:25.195695 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 12190
I20260924 07:06:25.195812 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:06:25.196130 3940 external_mini_cluster.cc:1428] Setting key a21c2aa16fe057d62abe1767bb64c37d
Sep 24 07:06:25 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233585, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.2@KRBTEST.COM" created.
I20260924 07:06:25.209389 12167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:25.194725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:41013 (local address 127.3.217.62:42643)
0924 07:06:25.194880 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:25.194885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:25.196105 (+ 1220us) server_negotiation.cc:407] Connection header received
0924 07:06:25.197514 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:25.197517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:25.197577 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:25.197644 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:25.198909 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:25.199755 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:25.201590 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:25.201900 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:25.201992 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:25.204962 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:25.204965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:25.204968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:25.204991 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:25.206738 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:25.207437 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:25.207443 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:25.207444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:25.207656 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:25.208239 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:25.208242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:25.208243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:25.208442 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:25.208538 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:25.209095 (+ 557us) server_negotiation.cc:299] Negotiation successful
0924 07:06:25.209210 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 07:06:25.210211 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:25.195084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.1:41013)
0924 07:06:25.195912 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:25.196000 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:25.197305 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:25.197888 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:25.197896 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:25.198270 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:25.198737 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:25.198750 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:25.199899 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:25.199903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:25.201314 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:25.201325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:25.202066 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:25.202070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:25.202263 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:25.202874 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:25.202894 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:25.204743 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:25.206947 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:25.206952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:25.206963 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:25.207249 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:25.207825 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:25.207827 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:25.207828 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:25.207972 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:25.208593 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:25.208600 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:25.208833 (+ 233us) client_negotiation.cc:769] Sending connection context
0924 07:06:25.209022 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 07:06:25.209314 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":128,"threads_started":1}
I20260924 07:06:25.211350 12319 heartbeater.cc:347] Connected to a master server at 127.3.217.62:42643
I20260924 07:06:25.211632 12319 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:25.212144 12319 heartbeater.cc:511] Master 127.3.217.62:42643 requested a full tablet report, sending...
I20260924 07:06:25.213707 12128 ts_manager.cc:194] Registered new tserver with Master: 686d5f5aa64544a59b2cc83c0bd483d1 (127.3.217.1:41703)
I20260924 07:06:25.215207 12128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.1@KRBTEST.COM'} at 127.3.217.1:41013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
I20260924 07:06:25.228039 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42643
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:38419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:25.342751 12324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:25.343053 12324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:25.343101 12324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:25.348085 12324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:25.348143 12324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:25.348201 12324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:06:25.354475 12324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:38419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:25.355366 12324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:25.356321 12324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:25.366007 12332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:25.366097 12329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:25.366037 12330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:25.366432 12324 server_base.cc:1034] running on GCE node
I20260924 07:06:25.366844 12324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:25.367468 12324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:25.368687 12324 hybrid_clock.cc:648] HybridClock initialized: now 1790233585368637 us; error 62 us; skew 500 ppm
Sep 24 07:06:25 dist-test-slave-2p7j krb5kdc[12083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233585, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:25.371425 12324 init.cc:377] Logged in from keytab as kudu/127.3.217.2@KRBTEST.COM (short username kudu)
I20260924 07:06:25.372300 12324 webserver.cc:460] Webserver started at http://127.3.217.2:41949/ using document root <none> and password file <none>
I20260924 07:06:25.372824 12324 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:25.372886 12324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:25.373090 12324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:25.375070 12324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "dc755bfc4d654029bf72d369cadcc4d8"
format_stamp: "Formatted at 2026-09-24 07:06:25 on dist-test-slave-2p7j"
server_key: "7b5e4457006844c3b739718afc57420f"
server_key_iv: "a9c4975d68393f706b57c7f88b967cfb"
server_key_version: "encryptionkey@0"
I20260924 07:06:25.375602 12324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "dc755bfc4d654029bf72d369cadcc4d8"
format_stamp: "Formatted at 2026-09-24 07:06:25 on dist-test-slave-2p7j"
server_key: "7b5e4457006844c3b739718afc57420f"
server_key_iv: "a9c4975d68393f706b57c7f88b967cfb"
server_key_version: "encryptionkey@0"
I20260924 07:06:25.378998 12324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:06:25.381440 12339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:25.382673 12324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:06:25.382795 12324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "dc755bfc4d654029bf72d369cadcc4d8"
format_stamp: "Formatted at 2026-09-24 07:06:25 on dist-test-slave-2p7j"
server_key: "7b5e4457006844c3b739718afc57420f"
server_key_iv: "a9c4975d68393f706b57c7f88b967cfb"
server_key_version: "encryptionkey@0"
I20260924 07:06:25.382889 12324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:25.395707 12324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:25.398448 12324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:25.398607 12324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:25.399116 12324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:25.399900 12324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:25.399956 12324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:25.400008 12324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:25.400038 12324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:25.408849 12324 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:40125
I20260924 07:06:25.408969 12452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:40125 every 8 connection(s)
I20260924 07:06:25.409669 12324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:06:25.414211 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 12324
I20260924 07:06:25.414305 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:06:25.414651 3940 external_mini_cluster.cc:1428] Setting key 51746e7d2a426ee99d135ba0d67d6825
Sep 24 07:06:25 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233585, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.217.3@KRBTEST.COM; defaulting to no policy
I20260924 07:06:25.424201 12167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:25.411296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:54093 (local address 127.3.217.62:42643)
0924 07:06:25.411450 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:25.411453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:25.412308 (+ 855us) server_negotiation.cc:407] Connection header received
0924 07:06:25.413606 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:25.413609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:25.413662 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:25.413732 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:25.414861 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:25.415374 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:25.416497 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:25.416710 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:25.416795 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:25.419887 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:25.419892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:25.419895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:25.419919 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:25.421382 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:25.422265 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:25.422268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:25.422270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:25.422356 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:25.422952 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:25.422955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:25.422957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:25.423177 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:25.423282 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:25.423907 (+ 625us) server_negotiation.cc:299] Negotiation successful
0924 07:06:25.424044 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 07:06:25.424983 12455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:25.411704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.2:54093)
0924 07:06:25.412149 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:25.412187 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:25.413401 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:25.413902 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:25.413908 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:25.414242 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:25.414729 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:25.414748 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:25.415495 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:25.415530 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:25.416372 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:25.416381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:25.416846 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:25.416850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:25.417008 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:25.417556 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:25.417575 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:25.419703 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:25.421617 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:25.421628 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:25.421647 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:25.422048 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:25.422534 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:25.422543 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:25.422545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:25.422818 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:25.423313 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:25.423318 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:25.423649 (+ 331us) client_negotiation.cc:769] Sending connection context
0924 07:06:25.423822 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 07:06:25.424149 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":110,"threads_started":1}
I20260924 07:06:25.426263 12453 heartbeater.cc:347] Connected to a master server at 127.3.217.62:42643
I20260924 07:06:25.426507 12453 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:25.426987 12453 heartbeater.cc:511] Master 127.3.217.62:42643 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.3@KRBTEST.COM" created.
I20260924 07:06:25.428071 12128 ts_manager.cc:194] Registered new tserver with Master: dc755bfc4d654029bf72d369cadcc4d8 (127.3.217.2:40125)
I20260924 07:06:25.428648 12128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.2@KRBTEST.COM'} at 127.3.217.2:54093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
I20260924 07:06:25.444240 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42643
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:38419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:25.551965 12458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:25.552203 12458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:25.552246 12458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:25.556234 12458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:25.556284 12458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:25.556344 12458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:06:25.561669 12458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:38419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:25.562651 12458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:25.563822 12458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:25.574337 12466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:25.574337 12464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:25.574379 12463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:25.574779 12458 server_base.cc:1034] running on GCE node
I20260924 07:06:25.575143 12458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:25.575798 12458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:25.576969 12458 hybrid_clock.cc:648] HybridClock initialized: now 1790233585576956 us; error 25 us; skew 500 ppm
Sep 24 07:06:25 dist-test-slave-2p7j krb5kdc[12083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233585, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:25.580325 12458 init.cc:377] Logged in from keytab as kudu/127.3.217.3@KRBTEST.COM (short username kudu)
I20260924 07:06:25.581142 12458 webserver.cc:460] Webserver started at http://127.3.217.3:33281/ using document root <none> and password file <none>
I20260924 07:06:25.581633 12458 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:25.581689 12458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:25.581857 12458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:25.584146 12458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "e9f9d7aa951b48189433d0859ef5746a"
format_stamp: "Formatted at 2026-09-24 07:06:25 on dist-test-slave-2p7j"
server_key: "88e731dcc8b5c1f00cf4780417bf0083"
server_key_iv: "41429e94bec91df0ca157d032300c832"
server_key_version: "encryptionkey@0"
I20260924 07:06:25.584805 12458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "e9f9d7aa951b48189433d0859ef5746a"
format_stamp: "Formatted at 2026-09-24 07:06:25 on dist-test-slave-2p7j"
server_key: "88e731dcc8b5c1f00cf4780417bf0083"
server_key_iv: "41429e94bec91df0ca157d032300c832"
server_key_version: "encryptionkey@0"
I20260924 07:06:25.589558 12458 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 07:06:25.592239 12473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:25.593804 12458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:06:25.593921 12458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "e9f9d7aa951b48189433d0859ef5746a"
format_stamp: "Formatted at 2026-09-24 07:06:25 on dist-test-slave-2p7j"
server_key: "88e731dcc8b5c1f00cf4780417bf0083"
server_key_iv: "41429e94bec91df0ca157d032300c832"
server_key_version: "encryptionkey@0"
I20260924 07:06:25.594038 12458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:25.604712 12458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:25.608307 12458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:25.608474 12458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:25.609037 12458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:25.609859 12458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:25.609995 12458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:25.610081 12458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:25.610117 12458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:25.620966 12458 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:45831
I20260924 07:06:25.620985 12586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:45831 every 8 connection(s)
I20260924 07:06:25.621963 12458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:06:25.630620 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 12458
I20260924 07:06:25.630753 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:06:25.631052 3940 external_mini_cluster.cc:1428] Setting key a2cd1bf6e29febda26de522e3d952aa9
Sep 24 07:06:25 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233585, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:06:25.638094 12167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:25.624055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:45297 (local address 127.3.217.62:42643)
0924 07:06:25.624191 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:25.624195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:25.625738 (+ 1543us) server_negotiation.cc:407] Connection header received
0924 07:06:25.627293 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:25.627297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:25.627365 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:25.627445 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:25.629066 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:25.629609 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:25.630732 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:25.630985 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:25.631076 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:25.634557 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:25.634560 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:25.634562 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:25.634583 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:25.635992 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:25.636458 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:25.636461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:25.636462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:25.636530 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:25.636836 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:25.636839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:25.636841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:25.637012 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:25.637245 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:25.637742 (+ 497us) server_negotiation.cc:299] Negotiation successful
0924 07:06:25.637856 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:06:25.638960 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:25.624789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.3:45297)
0924 07:06:25.625452 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:25.625510 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:25.627006 (+ 1496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:25.627717 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:25.627726 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:25.628149 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:25.628836 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:25.628867 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:25.629742 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:25.629745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:25.630603 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:25.630613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:25.631150 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:25.631156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:25.631368 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:25.632396 (+ 1028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:25.632419 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:25.634392 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:25.636108 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:25.636114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:25.636127 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:25.636357 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:25.636617 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:25.636620 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:25.636622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:25.636744 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:25.637257 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:25.637263 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:25.637498 (+ 235us) client_negotiation.cc:769] Sending connection context
0924 07:06:25.637682 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 07:06:25.638103 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":158,"threads_started":1}
I20260924 07:06:25.640329 12587 heartbeater.cc:347] Connected to a master server at 127.3.217.62:42643
I20260924 07:06:25.640805 12587 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:25.641464 12587 heartbeater.cc:511] Master 127.3.217.62:42643 requested a full tablet report, sending...
I20260924 07:06:25.642827 12128 ts_manager.cc:194] Registered new tserver with Master: e9f9d7aa951b48189433d0859ef5746a (127.3.217.3:45831)
I20260924 07:06:25.643368 12128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.3@KRBTEST.COM'} at 127.3.217.3:45297
I20260924 07:06:25.645920 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.1.keytab.
Entry for principal oryx/127.3.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.1.keytab.
WARNING: no policy specified for oryx/127.3.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.2.keytab.
Entry for principal oryx/127.3.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.2.keytab.
WARNING: no policy specified for oryx/127.3.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.3.keytab.
Entry for principal oryx/127.3.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.3.keytab.
I20260924 07:06:25.730517 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 12190
I20260924 07:06:25.735962 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 12324
I20260924 07:06:25.741431 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 12458
I20260924 07:06:25.746488 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 12096
I20260924 07:06:25.751629 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:42643
--webserver_interface=127.3.217.62
--webserver_port=39733
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:38419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:42643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:25.858291 12596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:25.858517 12596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:25.858553 12596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:25.862629 12596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:06:25.862690 12596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:25.862709 12596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:25.862725 12596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:06:25.862737 12596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:06:25.868008 12596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:38419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:42643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:42643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=39733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:25.868942 12596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:25.869860 12596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:25.877988 12604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:25.877988 12601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:25.878140 12602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:25.878211 12596 server_base.cc:1034] running on GCE node
I20260924 07:06:25.879256 12596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:25.880579 12596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:25.881769 12596 hybrid_clock.cc:648] HybridClock initialized: now 1790233585881739 us; error 43 us; skew 500 ppm
Sep 24 07:06:25 dist-test-slave-2p7j krb5kdc[12083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233585, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:25.884574 12596 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:06:25.885399 12596 webserver.cc:460] Webserver started at http://127.3.217.62:39733/ using document root <none> and password file <none>
I20260924 07:06:25.885919 12596 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:25.885989 12596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:25.889816 12596 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:06:25.892252 12611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:25.893589 12596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:06:25.893698 12596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "f46c1e1bcc0b4d1cb01fabc3b2f4dbb9"
format_stamp: "Formatted at 2026-09-24 07:06:24 on dist-test-slave-2p7j"
server_key: "4dff7502992299aee5bcadd29baac952"
server_key_iv: "21394c35d2323f3493238ef35975822f"
server_key_version: "encryptionkey@0"
I20260924 07:06:25.894222 12596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:25.909981 12596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:25.912873 12596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:25.913030 12596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:25.919410 12596 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:42643
I20260924 07:06:25.919438 12664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:42643 every 8 connection(s)
I20260924 07:06:25.920342 12596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:06:25.924263 12665 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:06:25.925757 12665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9: Bootstrap starting.
I20260924 07:06:25.927171 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 12596
I20260924 07:06:25.927665 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:41703
--local_ip_for_outbound_sockets=127.3.217.1
--tserver_master_addrs=127.3.217.62:42643
--webserver_port=45931
--webserver_interface=127.3.217.1
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:38419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.217.1
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:06:25.942158 12665 log.cc:824] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:25.950268 12665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:06:25.950634 12665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9: Bootstrap complete.
I20260924 07:06:25.954959 12665 raft_consensus.cc:348] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46c1e1bcc0b4d1cb01fabc3b2f4dbb9" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42643 } }
I20260924 07:06:25.955343 12665 raft_consensus.cc:729] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46c1e1bcc0b4d1cb01fabc3b2f4dbb9, State: Initialized, Role: FOLLOWER
I20260924 07:06:25.955813 12665 consensus_queue.cc:260] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46c1e1bcc0b4d1cb01fabc3b2f4dbb9" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42643 } }
I20260924 07:06:25.955928 12665 raft_consensus.cc:388] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:06:25.955989 12665 raft_consensus.cc:482] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:06:25.956073 12665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:06:25.958091 12665 raft_consensus.cc:504] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46c1e1bcc0b4d1cb01fabc3b2f4dbb9" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42643 } }
I20260924 07:06:25.958398 12665 leader_election.cc:302] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f46c1e1bcc0b4d1cb01fabc3b2f4dbb9; no voters:
I20260924 07:06:25.958669 12665 leader_election.cc:288] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 07:06:25.958794 12669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 07:06:25.959019 12669 raft_consensus.cc:686] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [term 2 LEADER]: Becoming Leader. State: Replica: f46c1e1bcc0b4d1cb01fabc3b2f4dbb9, State: Running, Role: LEADER
I20260924 07:06:25.959337 12669 consensus_queue.cc:237] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46c1e1bcc0b4d1cb01fabc3b2f4dbb9" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42643 } }
I20260924 07:06:25.959816 12665 sys_catalog.cc:564] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:06:25.960652 12670 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f46c1e1bcc0b4d1cb01fabc3b2f4dbb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46c1e1bcc0b4d1cb01fabc3b2f4dbb9" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42643 } } }
I20260924 07:06:25.960772 12670 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:25.961063 12671 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f46c1e1bcc0b4d1cb01fabc3b2f4dbb9. Latest consensus state: current_term: 2 leader_uuid: "f46c1e1bcc0b4d1cb01fabc3b2f4dbb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46c1e1bcc0b4d1cb01fabc3b2f4dbb9" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42643 } } }
I20260924 07:06:25.961156 12671 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:25.961472 12675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:06:25.964635 12675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:06:25.965255 12675 catalog_manager.cc:1259] Loaded cluster ID: a5e0e2d0c6cd4750b50b4ccb2bb42a33
I20260924 07:06:25.965312 12675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:06:25.967612 12675 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:06:25.968232 12675 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f46c1e1bcc0b4d1cb01fabc3b2f4dbb9: Loaded TSK: 0
I20260924 07:06:25.969401 12675 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:06:26.037145 12667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:26.037380 12667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:26.037421 12667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:26.041502 12667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:26.041560 12667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:26.041616 12667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:06:26.046721 12667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:38419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.1.keytab
--principal=oryx/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:41703
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=45931
--tserver_master_addrs=127.3.217.62:42643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:26.047660 12667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:26.048578 12667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:26.058929 12694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:26.059041 12691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:26.059077 12692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:26.060063 12667 server_base.cc:1034] running on GCE node
I20260924 07:06:26.060572 12667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:26.061369 12667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:26.062559 12667 hybrid_clock.cc:648] HybridClock initialized: now 1790233586062540 us; error 33 us; skew 500 ppm
Sep 24 07:06:26 dist-test-slave-2p7j krb5kdc[12083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233586, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:26.065955 12667 init.cc:377] Logged in from keytab as oryx/127.3.217.1@KRBTEST.COM (short username oryx)
I20260924 07:06:26.067013 12667 webserver.cc:460] Webserver started at http://127.3.217.1:45931/ using document root <none> and password file <none>
I20260924 07:06:26.067732 12667 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:26.067845 12667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:26.073294 12667 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 07:06:26.075923 12701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:26.077523 12667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 07:06:26.077644 12667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "686d5f5aa64544a59b2cc83c0bd483d1"
format_stamp: "Formatted at 2026-09-24 07:06:25 on dist-test-slave-2p7j"
server_key: "8836008b45ca7dfc00943d4d914ee957"
server_key_iv: "2e7281841ed56b7af6c214e89103657a"
server_key_version: "encryptionkey@0"
I20260924 07:06:26.078620 12667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:26.109964 12667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:26.113547 12667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:26.113736 12667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:26.114398 12667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:26.115350 12667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:26.115414 12667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:26.115466 12667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:26.115495 12667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:26.125516 12667 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:41703
I20260924 07:06:26.125572 12815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:41703 every 8 connection(s)
I20260924 07:06:26.126447 12667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:06:26.135071 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 12667
I20260924 07:06:26.135569 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:40125
--local_ip_for_outbound_sockets=127.3.217.2
--tserver_master_addrs=127.3.217.62:42643
--webserver_port=41949
--webserver_interface=127.3.217.2
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:38419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.217.2
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:06:26 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:26 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:26.140622 12819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:26.128584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.1:45365)
0924 07:06:26.129038 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:26.129092 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:26.130787 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:26.132207 (+ 1420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:26.132216 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:26.132622 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:26.133090 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:26.133105 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.134328 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.134332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:26.135163 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:26.135172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.136153 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.136156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:26.136294 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:26.136849 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:26.136864 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:26.139911 (+ 3047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":122,"threads_started":1}
W20260924 07:06:26.140985 12818 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:26.128483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:45365 (local address 127.3.217.62:42643)
0924 07:06:26.128967 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:26.128986 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:26.129330 (+ 344us) server_negotiation.cc:407] Connection header received
0924 07:06:26.131394 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:26.131407 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:26.131764 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:26.132070 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:26.133267 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.134193 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.135295 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.135976 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.136178 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:26.140177 (+ 3999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.1:45365: BlockingRecv error: recv got EOF from 127.3.217.1:45365 (error 108)
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":112,"threads_started":1}
W20260924 07:06:26.141183 12816 heartbeater.cc:644] Failed to heartbeat to 127.3.217.62:42643 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.217.62:42643: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Sep 24 07:06:26 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:26 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:26.147404 12819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:26.141603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.1:45689)
0924 07:06:26.141758 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:26.141772 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:26.141837 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:26.142043 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:26.142045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:26.142167 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:26.142251 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:26.142256 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.143262 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.143265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:26.143865 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:26.143874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.144444 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.144447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:26.144517 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:26.144962 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:26.144973 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:26.147266 (+ 2293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94}
W20260924 07:06:26.147553 12818 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:26.141709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:45689 (local address 127.3.217.62:42643)
0924 07:06:26.141829 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:26.141833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:26.141847 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:06:26.141892 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:26.141895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:26.141941 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:26.142035 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:26.142357 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.143099 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.143982 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.144319 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.144424 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:26.147419 (+ 2995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.1:45689: BlockingRecv error: recv got EOF from 127.3.217.1:45689 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 07:06:26 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:26 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:26.154323 12819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:26.148231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.1:55605)
0924 07:06:26.148445 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:26.148459 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:26.148540 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:26.148750 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:26.148753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:26.148908 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:26.148991 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:26.148996 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.149920 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.149923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:26.150777 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:26.150786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.151401 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.151404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:26.151483 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:26.151903 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:26.151917 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:26.154196 (+ 2279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 07:06:26.154515 12816 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:06:26.154685 12818 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:26.148266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:55605 (local address 127.3.217.62:42643)
0924 07:06:26.148359 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:26.148362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:26.148526 (+ 164us) server_negotiation.cc:407] Connection header received
0924 07:06:26.148615 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:26.148617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:26.148654 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:26.148719 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:26.149192 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.149823 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.150956 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.151248 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.151375 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:26.154400 (+ 3025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.1:55605: BlockingRecv error: recv got EOF from 127.3.217.1:55605 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260924 07:06:26.250162 12820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:26.250416 12820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:26.250473 12820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:26.255165 12820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:26.255241 12820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:26.255323 12820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:06:26.261114 12820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:38419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.2.keytab
--principal=oryx/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:40125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=41949
--tserver_master_addrs=127.3.217.62:42643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:26.262144 12820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:26.263144 12820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:26.272966 12826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:26.272989 12825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:26.272948 12828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:26.273384 12820 server_base.cc:1034] running on GCE node
I20260924 07:06:26.273876 12820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:26.274904 12820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:26.276242 12820 hybrid_clock.cc:648] HybridClock initialized: now 1790233586276240 us; error 64 us; skew 500 ppm
Sep 24 07:06:26 dist-test-slave-2p7j krb5kdc[12083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233586, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:26.279628 12820 init.cc:377] Logged in from keytab as oryx/127.3.217.2@KRBTEST.COM (short username oryx)
I20260924 07:06:26.280531 12820 webserver.cc:460] Webserver started at http://127.3.217.2:41949/ using document root <none> and password file <none>
I20260924 07:06:26.281071 12820 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:26.281150 12820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:26.285118 12820 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:06:26.287665 12835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:26.289486 12820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:06:26.289695 12820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "dc755bfc4d654029bf72d369cadcc4d8"
format_stamp: "Formatted at 2026-09-24 07:06:25 on dist-test-slave-2p7j"
server_key: "7b5e4457006844c3b739718afc57420f"
server_key_iv: "a9c4975d68393f706b57c7f88b967cfb"
server_key_version: "encryptionkey@0"
I20260924 07:06:26.290402 12820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:26.303634 12820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:26.306854 12820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:26.307044 12820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:26.307685 12820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:26.308509 12820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:26.308560 12820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:26.308619 12820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:26.308652 12820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:26.317909 12820 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:40125
I20260924 07:06:26.318020 12948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:40125 every 8 connection(s)
I20260924 07:06:26.318917 12820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:06:26.322908 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 12820
I20260924 07:06:26.323401 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:45831
--local_ip_for_outbound_sockets=127.3.217.3
--tserver_master_addrs=127.3.217.62:42643
--webserver_port=33281
--webserver_interface=127.3.217.3
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:38419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.217.3
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:06:26 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:26 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 07:06:26.332417 12818 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:26.320779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:58437 (local address 127.3.217.62:42643)
0924 07:06:26.321074 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:26.321079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:26.322043 (+ 964us) server_negotiation.cc:407] Connection header received
0924 07:06:26.323000 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:26.323004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:26.323064 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:26.323136 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:26.324627 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.325453 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.327174 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.327546 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.327655 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:26.332223 (+ 4568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.2:58437: BlockingRecv error: recv got EOF from 127.3.217.2:58437 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 07:06:26.332670 12951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:26.321272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.2:58437)
0924 07:06:26.321907 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:26.321946 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:26.322695 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:26.323326 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:26.323334 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:26.323848 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:26.324455 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:26.324473 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.325811 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.325816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:26.326961 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:26.326982 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.327800 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.327804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:26.328008 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:26.328966 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:26.328986 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:26.331905 (+ 2919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":115,"threads_started":1}
W20260924 07:06:26.333384 12949 heartbeater.cc:644] Failed to heartbeat to 127.3.217.62:42643 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.217.62:42643: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Sep 24 07:06:26 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:26 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:26.339233 12951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:26.333919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.2:55951)
0924 07:06:26.334054 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:26.334068 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:26.334141 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:26.334383 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:26.334386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:26.334538 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:26.334636 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:26.334641 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.335314 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.335318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:26.335924 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:26.335932 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.336320 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.336323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:26.336407 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:26.337010 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:26.337022 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:26.339114 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 07:06:26.339370 12818 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:26.334062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:55951 (local address 127.3.217.62:42643)
0924 07:06:26.334210 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:26.334214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:26.334226 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:06:26.334257 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:26.334259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:26.334290 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:26.334351 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:26.334742 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.335216 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.336026 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.336224 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.336306 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:26.339257 (+ 2951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.2:55951: BlockingRecv error: recv got EOF from 127.3.217.2:55951 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 07:06:26 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:26 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:26.345024 12951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:26.339883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.2:37737)
0924 07:06:26.340011 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:26.340025 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:26.340094 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:26.340286 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:26.340289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:26.340448 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:26.340530 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:26.340536 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.341195 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.341198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:26.342043 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:26.342052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.342574 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.342577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:26.342656 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:26.343087 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:26.343100 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:26.344906 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 07:06:26.345153 12818 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:26.340000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:37737 (local address 127.3.217.62:42643)
0924 07:06:26.340098 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:26.340101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:26.340113 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:06:26.340164 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:26.340166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:26.340196 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:26.340253 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:26.340642 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.341093 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.342171 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.342463 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.342561 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:26.345040 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.2:37737: BlockingRecv error: recv got EOF from 127.3.217.2:37737 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 07:06:26.345432 12949 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:06:26.436648 12952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:26.436885 12952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:26.436936 12952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:26.440779 12952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:26.440831 12952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:26.440886 12952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:06:26.445941 12952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:38419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/krb5kdc/oryx_127.3.217.3.keytab
--principal=oryx/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:45831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=33281
--tserver_master_addrs=127.3.217.62:42643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:26.446834 12952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:26.447793 12952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:26.456737 12958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:26.456741 12957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:26.457029 12960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:26.457124 12952 server_base.cc:1034] running on GCE node
I20260924 07:06:26.457646 12952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:26.458218 12952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:26.459486 12952 hybrid_clock.cc:648] HybridClock initialized: now 1790233586459446 us; error 37 us; skew 500 ppm
Sep 24 07:06:26 dist-test-slave-2p7j krb5kdc[12083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233586, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:26.463171 12952 init.cc:377] Logged in from keytab as oryx/127.3.217.3@KRBTEST.COM (short username oryx)
I20260924 07:06:26.464308 12952 webserver.cc:460] Webserver started at http://127.3.217.3:33281/ using document root <none> and password file <none>
I20260924 07:06:26.465034 12952 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:26.465119 12952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:26.469269 12952 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 07:06:26.471323 12967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:26.472618 12952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:06:26.472729 12952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "e9f9d7aa951b48189433d0859ef5746a"
format_stamp: "Formatted at 2026-09-24 07:06:25 on dist-test-slave-2p7j"
server_key: "88e731dcc8b5c1f00cf4780417bf0083"
server_key_iv: "41429e94bec91df0ca157d032300c832"
server_key_version: "encryptionkey@0"
I20260924 07:06:26.473234 12952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:26.484226 12952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:26.487591 12952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:26.487764 12952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:26.488330 12952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:26.489115 12952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:26.489162 12952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:26.489215 12952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:26.489244 12952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:26.498621 12952 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:45831
I20260924 07:06:26.498697 13080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:45831 every 8 connection(s)
I20260924 07:06:26.499567 12952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:06:26.500322 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 12952
Sep 24 07:06:26 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:26 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:26.510699 12818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:26.501138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49346 (local address 127.3.217.62:42643)
0924 07:06:26.501287 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:26.501291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:26.501483 (+ 192us) server_negotiation.cc:407] Connection header received
0924 07:06:26.501599 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:26.501602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:26.501645 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:26.501718 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:26.502589 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.503366 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.504288 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.504603 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.504701 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:26.506520 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:26.506527 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:26.506543 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:26.506574 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:26.508524 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:26.509036 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:26.509042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:26.509047 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:26.509187 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:26.509470 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:26.509474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:26.509475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:26.509841 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:26.509999 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:26.510320 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 07:06:26.510549 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 07:06:26.511617 13084 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:26.501747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:44453 (local address 127.3.217.62:42643)
0924 07:06:26.501960 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:26.501963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:26.502432 (+ 469us) server_negotiation.cc:407] Connection header received
0924 07:06:26.503973 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:26.503976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:26.504025 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:26.504105 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:26.505366 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.506145 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.507704 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.507926 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.508010 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:26.511481 (+ 3471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.3:44453: BlockingRecv error: recv got EOF from 127.3.217.3:44453 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":83,"threads_started":1}
I20260924 07:06:26.512498 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:26.501835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.3:44453)
0924 07:06:26.502300 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:26.502337 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:26.503781 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:26.504289 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:26.504297 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:26.504737 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:26.505226 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:26.505239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.506291 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.506294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:26.507578 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:26.507589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.508079 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.508083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:26.508265 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:26.508947 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:26.508965 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:26.511341 (+ 2376us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":144,"threads_started":1}
W20260924 07:06:26.513306 13081 heartbeater.cc:644] Failed to heartbeat to 127.3.217.62:42643 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.217.62:42643: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Sep 24 07:06:26 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:26 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:26.518635 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:26.513685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.3:40483)
0924 07:06:26.513855 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:26.513872 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:26.513958 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:26.514195 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:26.514198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:26.514356 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:26.514450 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:26.514456 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.515172 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.515175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:26.515747 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:26.515758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.516436 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.516440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:26.516538 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:26.516968 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:26.516985 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:26.518547 (+ 1562us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":72}
W20260924 07:06:26.518790 13084 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:26.513815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:40483 (local address 127.3.217.62:42643)
0924 07:06:26.513941 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:26.513946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:26.513960 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:06:26.514046 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:26.514049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:26.514096 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:26.514166 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:26.514568 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.515062 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.515883 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.516213 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.516321 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:26.518689 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.3:40483: BlockingRecv error: recv got EOF from 127.3.217.3:40483 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 07:06:26 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:26 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:26.524021 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:26.519321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.3:37865)
0924 07:06:26.519451 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:26.519466 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:26.519560 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:26.519767 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:26.519770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:26.519896 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:26.519956 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:26.519960 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.520662 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.520665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:26.521335 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:26.521344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.521771 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.521775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:26.521846 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:26.522344 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:26.522355 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:26.523903 (+ 1548us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 07:06:26.524190 13084 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:26.519366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:37865 (local address 127.3.217.62:42643)
0924 07:06:26.519535 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:26.519539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:26.519551 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:06:26.519641 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:26.519644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:26.519679 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:26.519741 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:26.520080 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.520560 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.521455 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:26.521675 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:26.521754 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:26.524016 (+ 2262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.3:37865: BlockingRecv error: recv got EOF from 127.3.217.3:37865 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260924 07:06:26.524353 13081 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 07:06:27 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:27 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:27.160825 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:27.155182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.1:44733)
0924 07:06:27.155462 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:27.155476 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:27.155591 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:27.155911 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:27.155914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:27.156087 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:27.156200 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:27.156207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:27.157031 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:27.157034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:27.157624 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:27.157633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:27.158108 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:27.158111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:27.158189 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:27.158669 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:27.158679 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:27.160666 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":84,"threads_started":1}
W20260924 07:06:27.160907 13087 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:27.155302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:44733 (local address 127.3.217.62:42643)
0924 07:06:27.155521 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:27.155525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:27.155542 (+ 17us) server_negotiation.cc:407] Connection header received
0924 07:06:27.155719 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:27.155721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:27.155767 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:27.155847 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:27.156359 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:27.156889 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:27.157740 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:27.157984 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:27.158062 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:27.160800 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.1:44733: BlockingRecv error: recv got EOF from 127.3.217.1:44733 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":54,"threads_started":1}
Sep 24 07:06:27 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:27 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:27.351787 13088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:27.345973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.2:52043)
0924 07:06:27.346214 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:27.346231 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:27.346428 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:27.346707 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:27.346710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:27.346901 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:27.346998 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:27.347004 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:27.347807 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:27.347811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:27.348365 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:27.348373 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:27.348791 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:27.348795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:27.348873 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:27.349422 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:27.349435 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:27.351631 (+ 2196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":70,"threads_started":1}
W20260924 07:06:27.351886 13087 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:27.346130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:52043 (local address 127.3.217.62:42643)
0924 07:06:27.346298 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:27.346301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:27.346318 (+ 17us) server_negotiation.cc:407] Connection header received
0924 07:06:27.346552 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:27.346555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:27.346605 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:27.346668 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:27.347112 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:27.347708 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:27.348479 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:27.348693 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:27.348774 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:27.351789 (+ 3015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.2:52043: BlockingRecv error: recv got EOF from 127.3.217.2:52043 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 07:06:27 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:27 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:27.530192 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:27.524900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.3:37159)
0924 07:06:27.525214 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:27.525228 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:27.525337 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:27.525577 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:27.525580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:27.525765 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:27.525851 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:27.525856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:27.526565 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:27.526568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:27.527128 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:27.527135 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:27.527563 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:27.527567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:27.527643 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:27.528120 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:27.528132 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:27.530038 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":91,"threads_started":1}
W20260924 07:06:27.530265 13087 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:27.525048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:37159 (local address 127.3.217.62:42643)
0924 07:06:27.525214 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:27.525218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:27.525308 (+ 90us) server_negotiation.cc:407] Connection header received
0924 07:06:27.525427 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:27.525430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:27.525475 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:27.525538 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:27.525966 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:27.526464 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:27.527239 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:27.527441 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:27.527546 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:27.530169 (+ 2623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.3:37159: BlockingRecv error: recv got EOF from 127.3.217.3:37159 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 07:06:28 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:28 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:28.168769 13090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:28.161585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.1:53401)
0924 07:06:28.161883 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:28.161897 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:28.162004 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:28.162513 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:28.162516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:28.162697 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:28.162792 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:28.162797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:28.163937 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:28.163941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:28.164814 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:28.164825 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:28.165369 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:28.165373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:28.165438 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:28.166180 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:28.166192 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:28.168649 (+ 2457us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":87,"threads_started":1}
W20260924 07:06:28.168931 13091 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:28.161658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:53401 (local address 127.3.217.62:42643)
0924 07:06:28.161902 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:28.161905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:28.161952 (+ 47us) server_negotiation.cc:407] Connection header received
0924 07:06:28.162225 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:28.162231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:28.162296 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:28.162392 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:28.162924 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:28.163705 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:28.164993 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:28.165231 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:28.165330 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:28.168801 (+ 3471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.1:53401: BlockingRecv error: recv got EOF from 127.3.217.1:53401 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":70,"threads_started":1}
Sep 24 07:06:28 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:28 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:28.359820 13092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:28.352858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.2:56689)
0924 07:06:28.353249 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:28.353263 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:28.353382 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:28.353615 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:28.353618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:28.353816 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:28.354061 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:28.354080 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:28.354990 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:28.354994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:28.355714 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:28.355724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:28.356428 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:28.356432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:28.356542 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:28.357144 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:28.357156 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:28.359488 (+ 2332us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":91,"threads_started":1}
W20260924 07:06:28.359818 13091 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:28.353011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:56689 (local address 127.3.217.62:42643)
0924 07:06:28.353212 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:28.353216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:28.353339 (+ 123us) server_negotiation.cc:407] Connection header received
0924 07:06:28.353468 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:28.353471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:28.353513 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:28.353576 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:28.354321 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:28.354865 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:28.355838 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:28.356232 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:28.356390 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:28.359704 (+ 3314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.2:56689: BlockingRecv error: recv got EOF from 127.3.217.2:56689 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 24 07:06:28 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:28 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:28.536406 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:28.530911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.3:46023)
0924 07:06:28.531297 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:28.531320 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:28.531434 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:28.531707 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:28.531710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:28.531892 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:28.531993 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:28.531998 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:28.532741 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:28.532744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:28.533322 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:28.533331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:28.533763 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:28.533765 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:28.533836 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:28.534304 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:28.534336 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:28.536233 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":73,"threads_started":1}
W20260924 07:06:28.536458 13091 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:28.531191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:46023 (local address 127.3.217.62:42643)
0924 07:06:28.531304 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:28.531307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:28.531492 (+ 185us) server_negotiation.cc:407] Connection header received
0924 07:06:28.531555 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:28.531559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:28.531606 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:28.531673 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:28.532113 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:28.532637 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:28.533439 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:28.533656 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:28.533739 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:28.536353 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.3:46023: BlockingRecv error: recv got EOF from 127.3.217.3:46023 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 07:06:29 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:29 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:29.175433 13094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:29.169848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.1:38717)
0924 07:06:29.170208 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:29.170229 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:29.170412 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:29.170936 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:29.170940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:29.171148 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:29.171251 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:29.171258 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:29.172059 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:29.172062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:29.172616 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:29.172624 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:29.173050 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:29.173053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:29.173130 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:29.173608 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:29.173619 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:29.175270 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":96,"threads_started":1}
W20260924 07:06:29.175539 13095 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:29.170047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:38717 (local address 127.3.217.62:42643)
0924 07:06:29.170276 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:29.170279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:29.170356 (+ 77us) server_negotiation.cc:407] Connection header received
0924 07:06:29.170633 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:29.170637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:29.170698 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:29.170912 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:29.171380 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:29.171955 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:29.172732 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:29.172947 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:29.173028 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:29.175401 (+ 2373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.1:38717: BlockingRecv error: recv got EOF from 127.3.217.1:38717 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":62,"threads_started":1}
Sep 24 07:06:29 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:29 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:29.368382 13096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:29.360897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.2:57497)
0924 07:06:29.361214 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:29.361229 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:29.361345 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:29.361768 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:29.361771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:29.362152 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:29.362283 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:29.362291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:29.363209 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:29.363212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:29.364028 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:29.364038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:29.364613 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:29.364618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:29.364736 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:29.365644 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:29.365659 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:29.368079 (+ 2420us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":89,"threads_started":1}
W20260924 07:06:29.368472 13095 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:29.361019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:57497 (local address 127.3.217.62:42643)
0924 07:06:29.361245 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:29.361249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:29.361293 (+ 44us) server_negotiation.cc:407] Connection header received
0924 07:06:29.361491 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:29.361495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:29.361560 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:29.361669 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:29.362459 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:29.363083 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:29.364193 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:29.364438 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:29.364552 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:29.368355 (+ 3803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.2:57497: BlockingRecv error: recv got EOF from 127.3.217.2:57497 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 24 07:06:29 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:29 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:29.543917 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:29.537470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.3:53415)
0924 07:06:29.537891 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:29.537908 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:29.538075 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:29.538397 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:29.538400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:29.538653 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:29.538770 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:29.538776 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:29.539606 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:29.539611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:29.540211 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:29.540219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:29.540678 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:29.540681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:29.540765 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:29.541344 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:29.541357 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:29.543696 (+ 2339us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":111,"threads_started":1}
W20260924 07:06:29.543932 13095 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:29.537643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:53415 (local address 127.3.217.62:42643)
0924 07:06:29.537857 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:29.537862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:29.538003 (+ 141us) server_negotiation.cc:407] Connection header received
0924 07:06:29.538215 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:29.538218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:29.538272 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:29.538345 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:29.538896 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:29.539475 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:29.540334 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:29.540552 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:29.540640 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:29.543820 (+ 3180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.3:53415: BlockingRecv error: recv got EOF from 127.3.217.3:53415 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 24 07:06:30 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:30 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:30.189318 13098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:30.176320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.1:47233)
0924 07:06:30.176600 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:30.176613 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:30.176699 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:30.176989 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:30.176992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:30.177194 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:30.177289 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:30.177294 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:30.178050 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:30.178052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:30.178626 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:30.178634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:30.179068 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:30.179071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:30.179143 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:30.186526 (+ 7383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:30.186541 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:30.189178 (+ 2637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":80,"threads_started":1}
W20260924 07:06:30.189451 13099 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:30.176422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:47233 (local address 127.3.217.62:42643)
0924 07:06:30.176657 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:30.176661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:30.176675 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:06:30.176767 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:30.176769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:30.176813 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:30.176896 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:30.177373 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:30.177899 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:30.178737 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:30.178953 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:30.179057 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:30.189315 (+ 10258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.1:47233: BlockingRecv error: recv got EOF from 127.3.217.1:47233 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
Sep 24 07:06:30 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:30 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:30.374994 13100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:30.369367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.2:50417)
0924 07:06:30.369635 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:30.369651 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:30.369757 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:30.370292 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:30.370294 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:30.370509 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:30.370622 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:30.370627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:30.371473 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:30.371476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:30.372112 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:30.372121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:30.372557 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:30.372559 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:30.372614 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:30.373135 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:30.373147 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:30.374874 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":83,"threads_started":1}
W20260924 07:06:30.375175 13099 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:30.369461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:50417 (local address 127.3.217.62:42643)
0924 07:06:30.369614 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:30.369618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:30.369715 (+ 97us) server_negotiation.cc:407] Connection header received
0924 07:06:30.370005 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:30.370010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:30.370111 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:30.370256 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:30.370768 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:30.371360 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:30.372239 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:30.372468 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:30.372553 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:30.375018 (+ 2465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.2:50417: BlockingRecv error: recv got EOF from 127.3.217.2:50417 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 07:06:30 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:30 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:30.551275 13101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:30.544972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.3:54281)
0924 07:06:30.545326 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:30.545343 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:30.545466 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:30.545742 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:30.545745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:30.545954 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:30.546082 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:30.546091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:30.547232 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:30.547236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:30.547982 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:30.547991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:30.548614 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:30.548618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:30.548710 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:30.549342 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:30.549354 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:30.551152 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":156,"threads_started":1}
W20260924 07:06:30.551445 13099 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:30.545041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:54281 (local address 127.3.217.62:42643)
0924 07:06:30.545281 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:30.545286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:30.545449 (+ 163us) server_negotiation.cc:407] Connection header received
0924 07:06:30.545546 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:30.545548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:30.545604 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:30.545675 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:30.546259 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:30.547093 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:30.548127 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:30.548373 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:30.548555 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:30.551320 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.3:54281: BlockingRecv error: recv got EOF from 127.3.217.3:54281 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 24 07:06:31 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:31 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:31.196508 13102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:31.190276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.1:34937)
0924 07:06:31.190636 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:31.190653 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:31.190753 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:31.191163 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:31.191167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:31.191403 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:31.191566 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:31.191574 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:31.192398 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:31.192402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:31.192998 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:31.193006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:31.193475 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:31.193479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:31.193558 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:31.194158 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:31.194171 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:31.196324 (+ 2153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":115,"threads_started":1}
W20260924 07:06:31.196609 13103 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:31.190444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:34937 (local address 127.3.217.62:42643)
0924 07:06:31.190828 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:31.190833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:31.190852 (+ 19us) server_negotiation.cc:407] Connection header received
0924 07:06:31.190906 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:31.190909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:31.190970 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:31.191075 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:31.191692 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:31.192285 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:31.193114 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:31.193359 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:31.193466 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:31.196484 (+ 3018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.1:34937: BlockingRecv error: recv got EOF from 127.3.217.1:34937 (error 108)
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":116,"threads_started":1}
Sep 24 07:06:31 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:31 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:31.382722 13104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:31.375896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.2:59827)
0924 07:06:31.376347 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:31.376364 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:31.376503 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:31.376938 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:31.376941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:31.377282 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:31.377442 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:31.377457 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:31.378499 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:31.378502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:31.379164 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:31.379173 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:31.379735 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:31.379739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:31.379818 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:31.380358 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:31.380370 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:31.382522 (+ 2152us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":165,"threads_started":1}
W20260924 07:06:31.382803 13103 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:31.376095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:59827 (local address 127.3.217.62:42643)
0924 07:06:31.376346 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:31.376351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:31.376450 (+ 99us) server_negotiation.cc:407] Connection header received
0924 07:06:31.376600 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:31.376603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:31.376806 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:31.376900 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:31.377656 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:31.378382 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:31.379286 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:31.379608 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:31.379708 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:31.382679 (+ 2971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.2:59827: BlockingRecv error: recv got EOF from 127.3.217.2:59827 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Sep 24 07:06:31 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:31 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:31.558637 13105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:31.552241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.3:49963)
0924 07:06:31.552577 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:31.552592 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:31.552753 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:31.553036 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:31.553038 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:31.553267 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:31.553392 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:31.553397 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:31.554365 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:31.554368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:31.555018 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:31.555028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:31.555610 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:31.555613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:31.555684 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:31.556284 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:31.556296 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:31.558481 (+ 2185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":111,"threads_started":1}
W20260924 07:06:31.558755 13103 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:31.552342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:49963 (local address 127.3.217.62:42643)
0924 07:06:31.552489 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:31.552493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:31.552733 (+ 240us) server_negotiation.cc:407] Connection header received
0924 07:06:31.552844 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:31.552846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:31.552896 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:31.552996 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:31.553558 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:31.554162 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:31.555163 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:31.555433 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:31.555594 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:31.558625 (+ 3031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.3:49963: BlockingRecv error: recv got EOF from 127.3.217.3:49963 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 07:06:32 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:32 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:32.203329 13106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:32.197336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.1:48773)
0924 07:06:32.197610 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:32.197624 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:32.197713 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:32.197926 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:32.197928 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:32.198103 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:32.198203 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:32.198208 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:32.199072 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:32.199075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:32.199704 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:32.199713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:32.200413 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:32.200416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:32.200480 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:32.201000 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:32.201011 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:32.203172 (+ 2161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":109,"threads_started":1}
W20260924 07:06:32.203460 13107 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:32.197404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:48773 (local address 127.3.217.62:42643)
0924 07:06:32.197659 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:32.197663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:32.197675 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:06:32.197789 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:32.197791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:32.197832 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:32.197904 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:32.198322 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:32.198949 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:32.199891 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:32.200262 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:32.200372 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:32.203327 (+ 2955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.1:48773: BlockingRecv error: recv got EOF from 127.3.217.1:48773 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
Sep 24 07:06:32 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:32 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:32.389050 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:32.383770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.2:34791)
0924 07:06:32.384039 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:32.384050 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:32.384157 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:32.384401 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:32.384404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:32.384628 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:32.384713 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:32.384718 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:32.385452 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:32.385455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:32.385989 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:32.385996 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:32.386402 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:32.386405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:32.386478 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:32.386940 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:32.386951 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:32.388924 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":99,"threads_started":1}
W20260924 07:06:32.389182 13107 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:32.383790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:34791 (local address 127.3.217.62:42643)
0924 07:06:32.383920 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:32.383925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:32.384120 (+ 195us) server_negotiation.cc:407] Connection header received
0924 07:06:32.384246 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:32.384249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:32.384295 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:32.384366 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:32.384835 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:32.385352 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:32.386096 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:32.386309 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:32.386390 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:32.389066 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.2:34791: BlockingRecv error: recv got EOF from 127.3.217.2:34791 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 07:06:32 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:32 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:32.565701 13109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:32.559643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.3:51765)
0924 07:06:32.560061 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:32.560075 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:32.560205 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:32.560537 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:32.560541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:32.560734 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:32.560835 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:32.560841 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:32.561699 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:32.561701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:32.562404 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:32.562414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:32.562831 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:32.562834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:32.562893 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:32.563372 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:32.563384 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:32.565578 (+ 2194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":224,"threads_started":1}
W20260924 07:06:32.565882 13107 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:32.559722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:51765 (local address 127.3.217.62:42643)
0924 07:06:32.559950 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:32.559955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:32.560247 (+ 292us) server_negotiation.cc:407] Connection header received
0924 07:06:32.560299 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:32.560302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:32.560352 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:32.560423 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:32.560975 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:32.561580 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:32.562518 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:32.562728 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:32.562812 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:32.565764 (+ 2952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.3:51765: BlockingRecv error: recv got EOF from 127.3.217.3:51765 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 24 07:06:33 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:33 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:33.210546 13110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:33.204405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.1:46039)
0924 07:06:33.204765 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:33.204786 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:33.204913 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:33.205181 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:33.205184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:33.205393 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:33.205524 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:33.205531 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:33.206462 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:33.206466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:33.207161 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:33.207176 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:33.207877 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:33.207887 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:33.207973 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:33.208563 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:33.208576 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:33.210420 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":81,"threads_started":1}
W20260924 07:06:33.210677 13111 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:33.204470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:46039 (local address 127.3.217.62:42643)
0924 07:06:33.204748 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:33.204751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:33.204862 (+ 111us) server_negotiation.cc:407] Connection header received
0924 07:06:33.205003 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:33.205006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:33.205060 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:33.205130 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:33.205649 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:33.206261 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:33.207334 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:33.207612 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:33.207791 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:33.210548 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.1:46039: BlockingRecv error: recv got EOF from 127.3.217.1:46039 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":85,"threads_started":1}
Sep 24 07:06:33 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:33 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:33.395809 13112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:33.389841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.2:56473)
0924 07:06:33.390188 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:33.390204 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:33.390306 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:33.390573 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:33.390576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:33.390798 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:33.390903 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:33.390909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:33.391856 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:33.391859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:33.392460 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:33.392469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:33.392902 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:33.392905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:33.392968 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:33.393525 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:33.393537 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:33.395642 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":72,"threads_started":1}
W20260924 07:06:33.395845 13111 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:33.390089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:56473 (local address 127.3.217.62:42643)
0924 07:06:33.390261 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:33.390266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:33.390280 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:06:33.390407 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:33.390411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:33.390468 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:33.390531 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:33.391029 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:33.391727 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:33.392563 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:33.392778 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:33.392857 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:33.395751 (+ 2894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.2:56473: BlockingRecv error: recv got EOF from 127.3.217.2:56473 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 07:06:33 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:33 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:33.578824 13113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:33.566459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.3:45757)
0924 07:06:33.566757 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:33.566773 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:33.566903 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:33.567199 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:33.567202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:33.567426 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:33.567568 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:33.567576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:33.568388 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:33.568391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:33.568964 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:33.569056 (+ 92us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:33.569494 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:33.569497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:33.569575 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:33.576484 (+ 6909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:33.576497 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:33.578704 (+ 2207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":81,"threads_started":1}
W20260924 07:06:33.579273 13111 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:33.566599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:45757 (local address 127.3.217.62:42643)
0924 07:06:33.566758 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:33.566762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:33.566845 (+ 83us) server_negotiation.cc:407] Connection header received
0924 07:06:33.566993 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:33.566996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:33.567047 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:33.567126 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:33.567677 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:33.568292 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:33.569178 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:33.569392 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:33.569485 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:33.579056 (+ 9571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.3:45757: BlockingRecv error: recv got EOF from 127.3.217.3:45757 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 24 07:06:34 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:34 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:34.216589 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:34.211266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.1:40623)
0924 07:06:34.211566 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:34.211582 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:34.211691 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:34.211964 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:34.211966 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:34.212127 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:34.212219 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:34.212224 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:34.212984 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:34.212987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:34.213810 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:34.213821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:34.214208 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:34.214211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:34.214272 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:34.214729 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:34.214741 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:34.216468 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":92,"threads_started":1}
W20260924 07:06:34.216871 13115 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:34.211411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:40623 (local address 127.3.217.62:42643)
0924 07:06:34.211647 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:34.211650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:34.211662 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:06:34.211813 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:34.211815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:34.211860 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:34.211936 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:34.212304 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:34.212865 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:34.213907 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:34.214125 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:34.214201 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:34.216769 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.1:40623: BlockingRecv error: recv got EOF from 127.3.217.1:40623 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":59,"threads_started":1}
Sep 24 07:06:34 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:34 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:34.402640 13116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:34.396595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.2:37483)
0924 07:06:34.396816 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:34.396828 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:34.396926 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:34.397207 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:34.397209 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:34.397376 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:34.397466 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:34.397471 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:34.398466 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:34.398469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:34.399195 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:34.399204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:34.399895 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:34.399898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:34.399963 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:34.400509 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:34.400520 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:34.402509 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":65,"threads_started":1}
W20260924 07:06:34.402779 13115 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:34.396663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:37483 (local address 127.3.217.62:42643)
0924 07:06:34.396809 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:34.396812 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:34.396886 (+ 74us) server_negotiation.cc:407] Connection header received
0924 07:06:34.397034 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:34.397037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:34.397088 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:34.397164 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:34.397578 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:34.398334 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:34.399343 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:34.399757 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:34.399863 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:34.402670 (+ 2807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.2:37483: BlockingRecv error: recv got EOF from 127.3.217.2:37483 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 07:06:34 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:34 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:34.585556 13117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:34.579654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.3:58815)
0924 07:06:34.579925 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:34.579939 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:34.580042 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:34.580251 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:34.580253 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:34.580440 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:34.580539 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:34.580545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:34.581397 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:34.581402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:34.582077 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:34.582091 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:34.582610 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:34.582613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:34.582757 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:34.583223 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:34.583235 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:34.585423 (+ 2188us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
W20260924 07:06:34.585647 13115 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:34.579760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:58815 (local address 127.3.217.62:42643)
0924 07:06:34.579879 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:34.579882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:34.580004 (+ 122us) server_negotiation.cc:407] Connection header received
0924 07:06:34.580111 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:34.580113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:34.580156 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:34.580219 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:34.580667 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:34.581205 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:34.582243 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:34.582470 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:34.582565 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:34.585560 (+ 2995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.3:58815: BlockingRecv error: recv got EOF from 127.3.217.3:58815 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 07:06:35 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:35 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:35.230569 13118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:35.217251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.1:44375)
0924 07:06:35.217517 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:35.217530 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:35.217613 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:35.217814 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:35.217816 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:35.217981 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:35.218074 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:35.218078 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:35.218768 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:35.218770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:35.219312 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:35.219319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:35.219817 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:35.219821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:35.219898 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:35.228444 (+ 8546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:35.228456 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:35.230453 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
W20260924 07:06:35.230723 13119 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:35.217362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:44375 (local address 127.3.217.62:42643)
0924 07:06:35.217582 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:35.217585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:35.217598 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:06:35.217685 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:35.217687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:35.217727 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:35.217793 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:35.218154 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:35.218677 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:35.219439 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:35.219710 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:35.219793 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:35.230599 (+ 10806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.1:44375: BlockingRecv error: recv got EOF from 127.3.217.1:44375 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":63,"threads_started":1}
Sep 24 07:06:35 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:35 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 07:06:35.409133 13119 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:35.403611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:42107 (local address 127.3.217.62:42643)
0924 07:06:35.403749 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:35.403754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:35.403847 (+ 93us) server_negotiation.cc:407] Connection header received
0924 07:06:35.403950 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:35.403952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:35.403998 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:35.404064 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:35.404470 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:35.405013 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:35.405798 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:35.406035 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:35.406121 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:35.409034 (+ 2913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.2:42107: BlockingRecv error: recv got EOF from 127.3.217.2:42107 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:06:35.409067 13120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:35.403527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.2:42107)
0924 07:06:35.403781 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:35.403793 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:35.403886 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:35.404093 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:35.404095 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:35.404264 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:35.404352 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:35.404357 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:35.405126 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:35.405129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:35.405673 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:35.405680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:35.406142 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:35.406145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:35.406205 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:35.406655 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:35.406666 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:35.408914 (+ 2248us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":97,"threads_started":1}
Sep 24 07:06:35 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:35 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:35.591645 13121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:35.586548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.3:55707)
0924 07:06:35.586917 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:35.586932 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:35.587033 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:35.587244 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:35.587246 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:35.587417 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:35.587526 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:35.587531 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:35.588240 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:35.588243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:35.588809 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:35.588817 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:35.589268 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:35.589271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:35.589331 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:35.589794 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:35.589805 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:35.591475 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":169,"threads_started":1}
W20260924 07:06:35.591784 13119 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:35.586638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:55707 (local address 127.3.217.62:42643)
0924 07:06:35.586759 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:35.586763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:35.586997 (+ 234us) server_negotiation.cc:407] Connection header received
0924 07:06:35.587115 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:35.587117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:35.587161 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:35.587216 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:35.587609 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:35.588129 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:35.588925 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:35.589161 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:35.589240 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:35.591657 (+ 2417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.3:55707: BlockingRecv error: recv got EOF from 127.3.217.3:55707 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 07:06:36 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:36 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:36.236867 13122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:36.231415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.1:37081)
0924 07:06:36.231725 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:36.231744 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:36.231875 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:36.232140 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:36.232143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:36.232350 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:36.232456 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:36.232462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:36.233219 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:36.233222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:36.233778 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:36.233819 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:36.234279 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:36.234283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:36.234375 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:36.234888 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:36.234901 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:36.236706 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":130,"threads_started":1}
W20260924 07:06:36.236992 13123 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:36.231477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:37081 (local address 127.3.217.62:42643)
0924 07:06:36.231757 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:36.231761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:36.231825 (+ 64us) server_negotiation.cc:407] Connection header received
0924 07:06:36.231976 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:36.231979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:36.232029 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:36.232114 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:36.232571 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:36.233112 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:36.233940 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:36.234173 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:36.234267 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:36.236850 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.1:37081: BlockingRecv error: recv got EOF from 127.3.217.1:37081 (error 108)
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":74,"threads_started":1}
Sep 24 07:06:36 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:36 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:36.414939 13124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:36.409855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.2:36975)
0924 07:06:36.410108 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:36.410124 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:36.410226 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:36.410468 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:36.410470 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:36.410658 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:36.410745 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:36.410750 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:36.411459 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:36.411462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:36.412026 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:36.412033 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:36.412427 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:36.412430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:36.412499 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:36.413049 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:36.413060 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:36.414788 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":77,"threads_started":1}
W20260924 07:06:36.415037 13123 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:36.409935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:36975 (local address 127.3.217.62:42643)
0924 07:06:36.410108 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:36.410113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:36.410192 (+ 79us) server_negotiation.cc:407] Connection header received
0924 07:06:36.410318 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:36.410322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:36.410363 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:36.410438 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:36.410852 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:36.411360 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:36.412128 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:36.412329 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:36.412418 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:36.414918 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.2:36975: BlockingRecv error: recv got EOF from 127.3.217.2:36975 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 24 07:06:36 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:36 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:36.598428 13125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:36.592518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.3:35727)
0924 07:06:36.592844 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:36.592862 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:36.592968 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:36.593201 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:36.593204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:36.593390 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:36.593487 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:36.593492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:36.594207 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:36.594211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:36.594812 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:36.594820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:36.595224 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:36.595228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:36.595317 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:36.595900 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:36.595915 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:36.598308 (+ 2393us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":108,"threads_started":1}
W20260924 07:06:36.598666 13123 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:36.592755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:35727 (local address 127.3.217.62:42643)
0924 07:06:36.592918 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:36.592923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:36.592937 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:06:36.593054 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:36.593057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:36.593106 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:36.593168 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:36.593604 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:36.594114 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:36.594912 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:36.595128 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:36.595207 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:36.598499 (+ 3292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.3:35727: BlockingRecv error: recv got EOF from 127.3.217.3:35727 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 24 07:06:37 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:37 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:37.243623 13126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:37.237576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.1:50769)
0924 07:06:37.237823 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:37.237838 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:37.237937 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:37.238218 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:37.238221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:37.238400 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:37.238494 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:37.238499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:37.239354 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:37.239359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:37.240148 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:37.240167 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:37.240702 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:37.240706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:37.240799 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:37.241304 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:37.241316 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:37.243406 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":77,"threads_started":1}
W20260924 07:06:37.243702 13127 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:37.237651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:50769 (local address 127.3.217.62:42643)
0924 07:06:37.237895 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:37.237898 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:37.237911 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:06:37.238039 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:37.238041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:37.238093 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:37.238189 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:37.238605 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:37.239134 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:37.240315 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:37.240538 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:37.240650 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:37.243570 (+ 2920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.1:50769: BlockingRecv error: recv got EOF from 127.3.217.1:50769 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":68,"threads_started":1}
Sep 24 07:06:37 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:37 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:37.421978 13128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:37.415929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.2:39305)
0924 07:06:37.416173 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:37.416186 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:37.416282 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:37.416489 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:37.416491 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:37.416665 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:37.416752 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:37.416757 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:37.418148 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:37.418150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:37.418732 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:37.418740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:37.419244 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:37.419248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:37.419312 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:37.419935 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:37.419952 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:37.421782 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":79,"threads_started":1}
W20260924 07:06:37.422027 13127 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:37.416008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:39305 (local address 127.3.217.62:42643)
0924 07:06:37.416163 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:37.416168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:37.416247 (+ 79us) server_negotiation.cc:407] Connection header received
0924 07:06:37.416348 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:37.416350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:37.416391 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:37.416463 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:37.416865 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:37.417653 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:37.418869 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:37.419110 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:37.419207 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:37.421925 (+ 2718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.2:39305: BlockingRecv error: recv got EOF from 127.3.217.2:39305 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 07:06:37 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:37 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:37.605720 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:37.599410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.3:48943)
0924 07:06:37.599764 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:37.599780 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:37.599966 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:37.600344 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:37.600347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:37.600534 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:37.600636 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:37.600643 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:37.601430 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:37.601433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:37.601985 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:37.601993 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:37.602466 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:37.602469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:37.602533 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:37.603051 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:37.603062 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:37.605582 (+ 2520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":149,"threads_started":1}
W20260924 07:06:37.605820 13127 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:37.599587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:48943 (local address 127.3.217.62:42643)
0924 07:06:37.599761 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:37.599766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:37.599937 (+ 171us) server_negotiation.cc:407] Connection header received
0924 07:06:37.600055 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:37.600058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:37.600112 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:37.600192 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:37.600757 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:37.601308 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:37.602111 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:37.602333 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:37.602423 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:37.605726 (+ 3303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.3:48943: BlockingRecv error: recv got EOF from 127.3.217.3:48943 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 07:06:38 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:38 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:38.251538 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:38.244731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.1:37411)
0924 07:06:38.245105 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:38.245121 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:38.245231 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:38.245615 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:38.245617 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:38.245804 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:38.245907 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:38.245912 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:38.246764 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:38.246767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:38.247312 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:38.247319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:38.247777 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:38.247780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:38.247841 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:38.248589 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:38.248608 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:38.251326 (+ 2718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":119,"threads_started":1}
W20260924 07:06:38.251622 13131 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:38.244848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:37411 (local address 127.3.217.62:42643)
0924 07:06:38.245136 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:38.245140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:38.245211 (+ 71us) server_negotiation.cc:407] Connection header received
0924 07:06:38.245400 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:38.245404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:38.245464 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:38.245567 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:38.246063 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:38.246620 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:38.247430 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:38.247667 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:38.247755 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:38.251465 (+ 3710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.1:37411: BlockingRecv error: recv got EOF from 127.3.217.1:37411 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":72,"threads_started":1}
Sep 24 07:06:38 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:38 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:38.427915 13132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:38.422705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.2:34603)
0924 07:06:38.422961 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:38.422972 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:38.423081 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:38.423325 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:38.423327 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:38.423531 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:38.423631 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:38.423636 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:38.424391 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:38.424394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:38.425022 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:38.425032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:38.425448 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:38.425457 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:38.425524 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:38.425991 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:38.426003 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:38.427800 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":88,"threads_started":1}
W20260924 07:06:38.428046 13131 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:38.422771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:34603 (local address 127.3.217.62:42643)
0924 07:06:38.422894 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:38.422898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:38.423027 (+ 129us) server_negotiation.cc:407] Connection header received
0924 07:06:38.423180 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:38.423183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:38.423237 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:38.423300 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:38.423749 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:38.424281 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:38.425134 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:38.425341 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:38.425415 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:38.427912 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.2:34603: BlockingRecv error: recv got EOF from 127.3.217.2:34603 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 07:06:38 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:38 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:38.612552 13133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:38.606493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.3:34593)
0924 07:06:38.606762 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:38.606774 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:38.606871 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:38.607210 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:38.607214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:38.607468 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:38.607632 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:38.607638 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:38.608485 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:38.608488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:38.609148 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:38.609157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:38.609636 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:38.609642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:38.609729 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:38.610422 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:38.610433 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:38.612408 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":87,"threads_started":1}
W20260924 07:06:38.612627 13131 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:38.606558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:34593 (local address 127.3.217.62:42643)
0924 07:06:38.606674 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:38.606677 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:38.606834 (+ 157us) server_negotiation.cc:407] Connection header received
0924 07:06:38.606932 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:38.606934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:38.606977 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:38.607044 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:38.607807 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:38.608359 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:38.609281 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:38.609498 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:38.609583 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:38.612532 (+ 2949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.3:34593: BlockingRecv error: recv got EOF from 127.3.217.3:34593 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 07:06:39 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:39 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:39.257855 13134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:39.252315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.1:43429)
0924 07:06:39.252566 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:39.252579 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:39.252677 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:39.252867 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:39.252869 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:39.253044 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:39.253135 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:39.253140 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:39.254186 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:39.254191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:39.254831 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:39.254843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:39.255383 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:39.255386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:39.255473 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:39.255970 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:39.255982 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:39.257730 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
W20260924 07:06:39.258282 13135 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:39.252353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:43429 (local address 127.3.217.62:42643)
0924 07:06:39.252570 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:39.252574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:39.252630 (+ 56us) server_negotiation.cc:407] Connection header received
0924 07:06:39.252742 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:39.252745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:39.252785 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:39.252856 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:39.253326 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:39.253971 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:39.254989 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:39.255240 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:39.255333 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:39.258024 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.1:43429: BlockingRecv error: recv got EOF from 127.3.217.1:43429 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":64,"threads_started":1}
Sep 24 07:06:39 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:39 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:39.434598 13136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:39.428633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.2:40061)
0924 07:06:39.428889 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:39.428904 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:39.429013 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:39.429241 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:39.429250 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:39.429625 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:39.429848 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:39.429860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:39.430843 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:39.430846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:39.431410 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:39.431417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:39.431844 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:39.431848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:39.431919 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:39.432450 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:39.432465 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:39.434439 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
W20260924 07:06:39.434684 13135 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:39.428800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:40061 (local address 127.3.217.62:42643)
0924 07:06:39.428926 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:39.428930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:39.428959 (+ 29us) server_negotiation.cc:407] Connection header received
0924 07:06:39.429103 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:39.429106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:39.429149 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:39.429211 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:39.429976 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:39.430670 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:39.431534 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:39.431752 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:39.431835 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:39.434578 (+ 2743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.2:40061: BlockingRecv error: recv got EOF from 127.3.217.2:40061 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 07:06:39 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:39 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:39.619807 13137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:39.613603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.3:39363)
0924 07:06:39.613941 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:39.613959 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:39.614112 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:39.614416 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:39.614419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:39.614659 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:39.614773 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:39.614779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:39.615584 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:39.615588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:39.616300 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:39.616310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:39.616773 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:39.616776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:39.616980 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:39.617493 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:39.617504 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:39.619661 (+ 2157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":91,"threads_started":1}
W20260924 07:06:39.619921 13135 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:39.613740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:39363 (local address 127.3.217.62:42643)
0924 07:06:39.613942 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:39.613948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:39.614050 (+ 102us) server_negotiation.cc:407] Connection header received
0924 07:06:39.614237 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:39.614240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:39.614304 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:39.614373 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:39.614894 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:39.615470 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:39.616439 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:39.616665 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:39.616754 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:39.619809 (+ 3055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.3:39363: BlockingRecv error: recv got EOF from 127.3.217.3:39363 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 24 07:06:40 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:40 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:40.273028 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:40.258727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.1:41225)
0924 07:06:40.259028 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:40.259042 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:40.259137 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:40.259730 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:40.259732 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:40.260006 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:40.260118 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:40.260125 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:40.261140 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:40.261143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:40.261794 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:40.261804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:40.262553 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:40.262558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:40.262669 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:40.271060 (+ 8391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:40.271075 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:40.272872 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":83,"threads_started":1}
W20260924 07:06:40.273137 13139 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:40.259025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:41225 (local address 127.3.217.62:42643)
0924 07:06:40.259394 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:40.259398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:40.259414 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:06:40.259468 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:40.259471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:40.259557 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:40.259676 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:40.260285 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:40.261023 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:40.261929 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:40.262390 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:40.262490 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:40.273023 (+ 10533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.1:41225: BlockingRecv error: recv got EOF from 127.3.217.1:41225 (error 108)
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":95,"threads_started":1}
Sep 24 07:06:40 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:40 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:40.440721 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:40.435342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.2:59441)
0924 07:06:40.435610 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:40.435626 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:40.435738 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:40.436099 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:40.436102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:40.436318 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:40.436421 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:40.436427 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:40.437208 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:40.437210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:40.437768 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:40.437775 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:40.438223 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:40.438226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:40.438286 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:40.438803 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:40.438814 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:40.440588 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":78,"threads_started":1}
W20260924 07:06:40.440814 13139 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:40.435404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:59441 (local address 127.3.217.62:42643)
0924 07:06:40.435561 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:40.435565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:40.435711 (+ 146us) server_negotiation.cc:407] Connection header received
0924 07:06:40.435855 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:40.435858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:40.435926 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:40.436069 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:40.436557 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:40.437078 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:40.437879 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:40.438098 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:40.438175 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:40.440716 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.2:59441: BlockingRecv error: recv got EOF from 127.3.217.2:59441 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 07:06:40 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:40 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.3@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:40.625653 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:40.620587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.3:55683)
0924 07:06:40.620892 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:40.620907 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:40.621056 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:40.621279 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:40.621281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:40.621508 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:40.621606 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:40.621612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:40.622317 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:40.622321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:40.622903 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:40.622911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:40.623305 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:40.623309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:40.623383 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:40.623916 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:40.623928 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:40.625531 (+ 1603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":87,"threads_started":1}
W20260924 07:06:40.626047 13139 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:40.620667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:55683 (local address 127.3.217.62:42643)
0924 07:06:40.620905 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:40.620910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:40.621008 (+ 98us) server_negotiation.cc:407] Connection header received
0924 07:06:40.621146 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:40.621148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:40.621190 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:40.621247 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:40.621705 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:40.622223 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:40.623010 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:40.623213 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:40.623292 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:40.625909 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.3:55683: BlockingRecv error: recv got EOF from 127.3.217.3:55683 (error 108)
Metrics: {"server-negotiator.queue_time_us":164}
Sep 24 07:06:41 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:41 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.1@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:41.280346 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:41.274003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.1:60243)
0924 07:06:41.274429 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:41.274443 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:41.274539 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:41.274895 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:41.274899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:41.275156 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:41.275307 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:41.275315 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:41.276177 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:41.276180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:41.276800 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:41.276809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:41.277258 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:41.277261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:41.277326 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:41.278054 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:41.278070 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:41.280186 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":82,"threads_started":1}
W20260924 07:06:41.280457 13143 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:41.274089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:60243 (local address 127.3.217.62:42643)
0924 07:06:41.274318 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:41.274322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:41.274533 (+ 211us) server_negotiation.cc:407] Connection header received
0924 07:06:41.274622 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:41.274625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:41.274670 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:41.274738 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:41.275474 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:41.276060 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:41.276901 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:41.277139 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:41.277221 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:41.280319 (+ 3098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.1:60243: BlockingRecv error: recv got EOF from 127.3.217.1:60243 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":54,"threads_started":1}
Sep 24 07:06:41 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:06:41 dist-test-slave-2p7j krb5kdc[12083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.217.2@KRBTEST.COM for oryx/127.3.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:06:41.446589 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:41.441527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42643 (local address 127.3.217.2:36775)
0924 07:06:41.441769 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:41.441784 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:41.441886 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:41.442137 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:41.442140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:41.442329 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:41.442414 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:41.442419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:41.443136 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:41.443139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:41.443751 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:41.443759 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:41.444163 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:41.444167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:41.444238 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:41.444702 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:41.444713 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:41.446436 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.217.62:42643: Server oryx/127.3.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":78,"threads_started":1}
W20260924 07:06:41.446672 13143 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:41.441636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:36775 (local address 127.3.217.62:42643)
0924 07:06:41.441773 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:41.441778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:41.441856 (+ 78us) server_negotiation.cc:407] Connection header received
0924 07:06:41.441972 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:41.441975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:41.442019 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:41.442090 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:41.442517 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:41.443040 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:41.443850 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:41.444072 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:41.444165 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:41.446575 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.217.2:36775: BlockingRecv error: recv got EOF from 127.3.217.2:36775 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 07:06:41.501900 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 12667
I20260924 07:06:41.506748 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 12820
I20260924 07:06:41.511358 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 12952
I20260924 07:06:41.515834 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 12596
2026-09-24T07:06:41Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17961 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T07:06:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:06:41Z Disabled control of system clock
I20260924 07:06:41.544196 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:44543
--webserver_interface=127.3.217.62
--webserver_port=0
--builtin_ntp_servers=127.3.217.20:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:44543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 07:06:41.649935 13148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:41.650198 13148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:41.650233 13148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:41.654117 13148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:06:41.654165 13148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:41.654179 13148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:41.654196 13148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:06:41.654209 13148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:06:41.658756 13148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:44543
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:44543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:41.659761 13148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:41.660988 13148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:41.670703 13153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:41.670766 13148 server_base.cc:1034] running on GCE node
W20260924 07:06:41.670823 13156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:41.670825 13154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:41.671519 13148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:41.673017 13148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:41.674224 13148 hybrid_clock.cc:648] HybridClock initialized: now 1790233601674190 us; error 43 us; skew 500 ppm
I20260924 07:06:41.676142 13148 webserver.cc:460] Webserver started at http://127.3.217.62:32911/ using document root <none> and password file <none>
I20260924 07:06:41.676681 13148 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:41.676748 13148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:41.676949 13148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:41.678912 13148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "cfcc718c14fe4fb69b2431cd52c42885"
format_stamp: "Formatted at 2026-09-24 07:06:41 on dist-test-slave-2p7j"
server_key: "8e38c637ab93414eb97d0766e6384d32"
server_key_iv: "55d2b0a7cae38f470010669f506ecd15"
server_key_version: "encryptionkey@0"
I20260924 07:06:41.679448 13148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "cfcc718c14fe4fb69b2431cd52c42885"
format_stamp: "Formatted at 2026-09-24 07:06:41 on dist-test-slave-2p7j"
server_key: "8e38c637ab93414eb97d0766e6384d32"
server_key_iv: "55d2b0a7cae38f470010669f506ecd15"
server_key_version: "encryptionkey@0"
I20260924 07:06:41.683215 13148 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:06:41.686300 13162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:41.687418 13148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:06:41.687572 13148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "cfcc718c14fe4fb69b2431cd52c42885"
format_stamp: "Formatted at 2026-09-24 07:06:41 on dist-test-slave-2p7j"
server_key: "8e38c637ab93414eb97d0766e6384d32"
server_key_iv: "55d2b0a7cae38f470010669f506ecd15"
server_key_version: "encryptionkey@0"
I20260924 07:06:41.687677 13148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:41.702248 13148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:41.703172 13148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:41.703351 13148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:41.709832 13148 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:44543
I20260924 07:06:41.709856 13214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:44543 every 8 connection(s)
I20260924 07:06:41.710704 13148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:06:41.713690 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:41.718683 13215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfcc718c14fe4fb69b2431cd52c42885: Bootstrap starting.
I20260924 07:06:41.720265 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 13148
I20260924 07:06:41.720362 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:06:41.720610 3940 external_mini_cluster.cc:1428] Setting key a412ec1d81b96b6493572d4ccc126718
I20260924 07:06:41.721243 13215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfcc718c14fe4fb69b2431cd52c42885: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:41.722076 13215 log.cc:824] T 00000000000000000000000000000000 P cfcc718c14fe4fb69b2431cd52c42885: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:41.724682 13215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfcc718c14fe4fb69b2431cd52c42885: No bootstrap required, opened a new log
I20260924 07:06:41.728781 13215 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfcc718c14fe4fb69b2431cd52c42885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcc718c14fe4fb69b2431cd52c42885" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 44543 } }
I20260924 07:06:41.729157 13215 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfcc718c14fe4fb69b2431cd52c42885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:41.729305 13215 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfcc718c14fe4fb69b2431cd52c42885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcc718c14fe4fb69b2431cd52c42885, State: Initialized, Role: FOLLOWER
I20260924 07:06:41.729846 13215 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfcc718c14fe4fb69b2431cd52c42885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcc718c14fe4fb69b2431cd52c42885" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 44543 } }
I20260924 07:06:41.730000 13215 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfcc718c14fe4fb69b2431cd52c42885 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:06:41.730073 13215 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfcc718c14fe4fb69b2431cd52c42885 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:06:41.730176 13215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfcc718c14fe4fb69b2431cd52c42885 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:41.730348 13218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:41.721652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60580 (local address 127.3.217.62:44543)
0924 07:06:41.722735 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:41.722761 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:41.722900 (+ 139us) server_negotiation.cc:407] Connection header received
0924 07:06:41.723753 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:41.723772 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:41.724084 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:41.724401 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:41.724817 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:41.725768 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:41.726588 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:41.727031 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:41.727191 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:41.728249 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:41.728254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:41.728265 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:06:41.728638 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:41.728692 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:06:41.728711 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:41.728793 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:41.728921 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 07:06:41.729177 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":115,"threads_started":1}
I20260924 07:06:41.731642 13215 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfcc718c14fe4fb69b2431cd52c42885 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcc718c14fe4fb69b2431cd52c42885" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 44543 } }
I20260924 07:06:41.731978 13215 leader_election.cc:302] T 00000000000000000000000000000000 P cfcc718c14fe4fb69b2431cd52c42885 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfcc718c14fe4fb69b2431cd52c42885; no voters:
I20260924 07:06:41.732249 13215 leader_election.cc:288] T 00000000000000000000000000000000 P cfcc718c14fe4fb69b2431cd52c42885 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:06:41.732342 13220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfcc718c14fe4fb69b2431cd52c42885 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:41.732542 13220 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfcc718c14fe4fb69b2431cd52c42885 [term 1 LEADER]: Becoming Leader. State: Replica: cfcc718c14fe4fb69b2431cd52c42885, State: Running, Role: LEADER
I20260924 07:06:41.732885 13220 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfcc718c14fe4fb69b2431cd52c42885 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcc718c14fe4fb69b2431cd52c42885" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 44543 } }
I20260924 07:06:41.733435 13215 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfcc718c14fe4fb69b2431cd52c42885 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:06:41.734706 13221 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfcc718c14fe4fb69b2431cd52c42885 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfcc718c14fe4fb69b2431cd52c42885. Latest consensus state: current_term: 1 leader_uuid: "cfcc718c14fe4fb69b2431cd52c42885" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcc718c14fe4fb69b2431cd52c42885" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 44543 } } }
I20260924 07:06:41.734753 13222 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfcc718c14fe4fb69b2431cd52c42885 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfcc718c14fe4fb69b2431cd52c42885" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcc718c14fe4fb69b2431cd52c42885" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 44543 } } }
I20260924 07:06:41.734965 13222 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfcc718c14fe4fb69b2431cd52c42885 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:41.735399 13221 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfcc718c14fe4fb69b2431cd52c42885 [sys.catalog]: This master's current role is: LEADER
W20260924 07:06:41.738216 13235 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cfcc718c14fe4fb69b2431cd52c42885: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:06:41.738277 13235 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:06:41.738354 13233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:06:41.738979 13233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:06:41.744298 13233 catalog_manager.cc:1347] Generated new cluster ID: b26604c184e24d6a9ee1497542f4c178
I20260924 07:06:41.744347 13233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:06:41.768620 13233 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:06:41.769599 13233 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:06:41.778718 13233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfcc718c14fe4fb69b2431cd52c42885: Generated new TSK 0
I20260924 07:06:41.779454 13233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:06:41.792222 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:44543
--builtin_ntp_servers=127.3.217.20:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 07:06:41.899340 13239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:41.899607 13239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:41.899642 13239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:41.903475 13239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:41.903548 13239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:41.903600 13239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:06:41.908169 13239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:44543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:41.909142 13239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:41.910399 13239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:41.919348 13244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:41.919387 13247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:41.919371 13245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:41.919590 13239 server_base.cc:1034] running on GCE node
I20260924 07:06:41.919994 13239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:41.920655 13239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:41.921838 13239 hybrid_clock.cc:648] HybridClock initialized: now 1790233601921823 us; error 54 us; skew 500 ppm
I20260924 07:06:41.923655 13239 webserver.cc:460] Webserver started at http://127.3.217.1:43973/ using document root <none> and password file <none>
I20260924 07:06:41.924181 13239 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:41.924245 13239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:41.924417 13239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:41.926456 13239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "d662a0368dce4f9296eb9ea2a66e7d6c"
format_stamp: "Formatted at 2026-09-24 07:06:41 on dist-test-slave-2p7j"
server_key: "4bd99c32bd2111df74fe4a1b17c97c05"
server_key_iv: "24952b3141de7e928f8c9f79a1994ef8"
server_key_version: "encryptionkey@0"
I20260924 07:06:41.927026 13239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "d662a0368dce4f9296eb9ea2a66e7d6c"
format_stamp: "Formatted at 2026-09-24 07:06:41 on dist-test-slave-2p7j"
server_key: "4bd99c32bd2111df74fe4a1b17c97c05"
server_key_iv: "24952b3141de7e928f8c9f79a1994ef8"
server_key_version: "encryptionkey@0"
I20260924 07:06:41.930434 13239 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260924 07:06:41.933148 13253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:41.934335 13239 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:06:41.934455 13239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "d662a0368dce4f9296eb9ea2a66e7d6c"
format_stamp: "Formatted at 2026-09-24 07:06:41 on dist-test-slave-2p7j"
server_key: "4bd99c32bd2111df74fe4a1b17c97c05"
server_key_iv: "24952b3141de7e928f8c9f79a1994ef8"
server_key_version: "encryptionkey@0"
I20260924 07:06:41.934549 13239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:41.953114 13239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:41.953775 13239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:41.953922 13239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:41.954452 13239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:41.955556 13239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:41.955612 13239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:41.955651 13239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:41.955665 13239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:41.964825 13239 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:34571
I20260924 07:06:41.964845 13366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:34571 every 8 connection(s)
I20260924 07:06:41.965839 13239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:06:41.968081 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 13239
I20260924 07:06:41.968176 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:06:41.968454 3940 external_mini_cluster.cc:1428] Setting key 61f3b618970b3bf55ed460313de3562f
I20260924 07:06:41.970145 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:44543
--builtin_ntp_servers=127.3.217.20:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 07:06:41.975250 13218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:41.967759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:39759 (local address 127.3.217.62:44543)
0924 07:06:41.967890 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:41.967895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:41.968925 (+ 1030us) server_negotiation.cc:407] Connection header received
0924 07:06:41.970227 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:41.970230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:41.970280 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:41.970347 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:41.971770 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:41.972299 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:41.973444 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:41.973662 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:41.973746 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:41.974803 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:41.974807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:41.974809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:06:41.974820 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:41.974837 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:06:41.974845 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:41.974901 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:41.975054 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 07:06:41.975112 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:06:41.976110 13369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:41.968083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:44543 (local address 127.3.217.1:39759)
0924 07:06:41.968781 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:41.968829 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:41.970029 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:41.970500 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:41.970506 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:41.970584 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:41.971427 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:41.971452 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:41.972438 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:41.972442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:41.973317 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:41.973328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:41.973793 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:41.973797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:41.973950 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:41.974555 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:06:41.974556 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:41.974594 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:06:41.974654 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:41.974912 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:41.974921 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:41.974927 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:06:41.975045 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 07:06:41.975312 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":119,"threads_started":1}
I20260924 07:06:41.977267 13367 heartbeater.cc:347] Connected to a master server at 127.3.217.62:44543
I20260924 07:06:41.977540 13367 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:41.978165 13367 heartbeater.cc:511] Master 127.3.217.62:44543 requested a full tablet report, sending...
I20260924 07:06:41.980178 13179 ts_manager.cc:194] Registered new tserver with Master: d662a0368dce4f9296eb9ea2a66e7d6c (127.3.217.1:34571)
I20260924 07:06:41.981679 13179 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.217.1:39759
W20260924 07:06:42.076221 13370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:42.076557 13370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:42.076597 13370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:42.081825 13370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:42.081902 13370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:42.081971 13370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:06:42.086972 13370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:44543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:42.088358 13370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:42.089692 13370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:06:42.099689 13370 server_base.cc:1034] running on GCE node
W20260924 07:06:42.099910 13376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:42.099944 13375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:42.099977 13378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:42.100437 13370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:42.101035 13370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:42.102312 13370 hybrid_clock.cc:648] HybridClock initialized: now 1790233602102259 us; error 49 us; skew 500 ppm
I20260924 07:06:42.104712 13370 webserver.cc:460] Webserver started at http://127.3.217.2:36535/ using document root <none> and password file <none>
I20260924 07:06:42.105294 13370 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:42.105365 13370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:42.105562 13370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:42.108141 13370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "9d43fea46e6b499697546c6d1ef06505"
format_stamp: "Formatted at 2026-09-24 07:06:42 on dist-test-slave-2p7j"
server_key: "cd754fb2be8f8b6ba46ac61b084f0ca7"
server_key_iv: "f851ef781fd11d2027dbc1f451d6b447"
server_key_version: "encryptionkey@0"
I20260924 07:06:42.109133 13370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "9d43fea46e6b499697546c6d1ef06505"
format_stamp: "Formatted at 2026-09-24 07:06:42 on dist-test-slave-2p7j"
server_key: "cd754fb2be8f8b6ba46ac61b084f0ca7"
server_key_iv: "f851ef781fd11d2027dbc1f451d6b447"
server_key_version: "encryptionkey@0"
I20260924 07:06:42.112939 13370 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260924 07:06:42.115630 13384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:42.116739 13370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:06:42.116846 13370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "9d43fea46e6b499697546c6d1ef06505"
format_stamp: "Formatted at 2026-09-24 07:06:42 on dist-test-slave-2p7j"
server_key: "cd754fb2be8f8b6ba46ac61b084f0ca7"
server_key_iv: "f851ef781fd11d2027dbc1f451d6b447"
server_key_version: "encryptionkey@0"
I20260924 07:06:42.116950 13370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:42.144686 13370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:42.145282 13370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:42.145422 13370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:42.145954 13370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:42.146798 13370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:42.146845 13370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:42.146883 13370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:42.146896 13370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:42.155614 13370 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:40075
I20260924 07:06:42.155642 13497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:40075 every 8 connection(s)
I20260924 07:06:42.156447 13370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:06:42.166141 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 13370
I20260924 07:06:42.166250 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:06:42.166291 13218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:42.158297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:55659 (local address 127.3.217.62:44543)
0924 07:06:42.158521 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:42.158534 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:42.159861 (+ 1327us) server_negotiation.cc:407] Connection header received
0924 07:06:42.161499 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:42.161501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:42.161557 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:42.161642 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:42.162587 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:42.163126 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:42.164269 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:42.164491 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:42.164586 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:42.165687 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:42.165691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:42.165694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:06:42.165714 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:42.165750 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:06:42.165762 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:42.165865 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:42.166102 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 07:06:42.166157 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 07:06:42.166535 3940 external_mini_cluster.cc:1428] Setting key e75f659894a5a1418e40ec312265268d
I20260924 07:06:42.167332 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:42.158677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:44543 (local address 127.3.217.2:55659)
0924 07:06:42.159648 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:42.159710 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:42.161247 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:42.161847 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:42.161856 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:42.161952 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:42.162437 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:42.162451 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:42.163241 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:42.163244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:42.164152 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:42.164163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:42.164638 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:42.164641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:42.164785 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:42.165342 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:06:42.165344 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:42.165381 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:06:42.165440 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:42.165904 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:42.165913 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:42.165920 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 07:06:42.166059 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 07:06:42.166360 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":110,"threads_started":1}
I20260924 07:06:42.168556 13498 heartbeater.cc:347] Connected to a master server at 127.3.217.62:44543
I20260924 07:06:42.168831 13498 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:42.168828 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:44543
--builtin_ntp_servers=127.3.217.20:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 07:06:42.169310 13498 heartbeater.cc:511] Master 127.3.217.62:44543 requested a full tablet report, sending...
I20260924 07:06:42.170265 13179 ts_manager.cc:194] Registered new tserver with Master: 9d43fea46e6b499697546c6d1ef06505 (127.3.217.2:40075)
I20260924 07:06:42.170789 13179 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.217.2:55659
W20260924 07:06:42.283833 13501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:42.284076 13501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:42.284112 13501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:42.288110 13501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:42.288170 13501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:42.288223 13501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:06:42.292898 13501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:44543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:42.293881 13501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:42.295194 13501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:42.304998 13506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:42.305054 13507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:42.305060 13501 server_base.cc:1034] running on GCE node
W20260924 07:06:42.305016 13509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:42.305615 13501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:42.306176 13501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:42.307408 13501 hybrid_clock.cc:648] HybridClock initialized: now 1790233602307348 us; error 58 us; skew 500 ppm
I20260924 07:06:42.309705 13501 webserver.cc:460] Webserver started at http://127.3.217.3:44235/ using document root <none> and password file <none>
I20260924 07:06:42.310226 13501 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:42.310292 13501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:42.310458 13501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:42.312330 13501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "8a7dee3c3a2a4830a0536ddc17c8a2e8"
format_stamp: "Formatted at 2026-09-24 07:06:42 on dist-test-slave-2p7j"
server_key: "377e59705af1729fe472bf195a3821a7"
server_key_iv: "c8d61a714228ec4e26e0242391372150"
server_key_version: "encryptionkey@0"
I20260924 07:06:42.312825 13501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "8a7dee3c3a2a4830a0536ddc17c8a2e8"
format_stamp: "Formatted at 2026-09-24 07:06:42 on dist-test-slave-2p7j"
server_key: "377e59705af1729fe472bf195a3821a7"
server_key_iv: "c8d61a714228ec4e26e0242391372150"
server_key_version: "encryptionkey@0"
I20260924 07:06:42.316007 13501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 07:06:42.318172 13515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:42.319240 13501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:06:42.319427 13501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "8a7dee3c3a2a4830a0536ddc17c8a2e8"
format_stamp: "Formatted at 2026-09-24 07:06:42 on dist-test-slave-2p7j"
server_key: "377e59705af1729fe472bf195a3821a7"
server_key_iv: "c8d61a714228ec4e26e0242391372150"
server_key_version: "encryptionkey@0"
I20260924 07:06:42.319584 13501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:42.340510 13501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:42.341159 13501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:42.341337 13501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:42.341898 13501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:42.342813 13501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:42.342862 13501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:42.342908 13501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:42.342945 13501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:42.351292 13501 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:46027
I20260924 07:06:42.351321 13629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:46027 every 8 connection(s)
I20260924 07:06:42.352114 13501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:06:42.355082 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 13501
I20260924 07:06:42.355175 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:06:42.355391 3940 external_mini_cluster.cc:1428] Setting key 1d54735a70db58b5ce58953370120b8d
I20260924 07:06:42.361661 13218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:42.353773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:58169 (local address 127.3.217.62:44543)
0924 07:06:42.353925 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:42.353929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:42.354871 (+ 942us) server_negotiation.cc:407] Connection header received
0924 07:06:42.356535 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:42.356539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:42.356607 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:42.356697 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:42.358097 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:42.358640 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:42.359817 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:42.360046 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:42.360138 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:42.361218 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:42.361222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:42.361225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:06:42.361238 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:42.361258 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:06:42.361266 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:42.361324 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:42.361454 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 07:06:42.361524 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 07:06:42.362435 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:42.354077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:44543 (local address 127.3.217.3:58169)
0924 07:06:42.354721 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:42.354759 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:42.356256 (+ 1497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:42.357019 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:42.357033 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:42.357181 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:42.357931 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:42.357950 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:42.358796 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:42.358799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:42.359703 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:42.359714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:42.360192 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:42.360196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:42.360343 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:42.360878 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:06:42.360879 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:42.360918 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:06:42.361006 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:42.361334 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:42.361343 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:42.361349 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:06:42.361450 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 07:06:42.361708 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":111,"threads_started":1}
I20260924 07:06:42.363538 13630 heartbeater.cc:347] Connected to a master server at 127.3.217.62:44543
I20260924 07:06:42.363747 13630 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:42.364181 13630 heartbeater.cc:511] Master 127.3.217.62:44543 requested a full tablet report, sending...
I20260924 07:06:42.365113 13179 ts_manager.cc:194] Registered new tserver with Master: 8a7dee3c3a2a4830a0536ddc17c8a2e8 (127.3.217.3:46027)
I20260924 07:06:42.365651 13179 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.217.3:58169
I20260924 07:06:42.369402 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 07:06:42.371961 13218 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:42.371123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60582 (local address 127.3.217.62:44543)
0924 07:06:42.371228 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:42.371231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:42.371349 (+ 118us) server_negotiation.cc:407] Connection header received
0924 07:06:42.371439 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:42.371441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:42.371481 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:42.371596 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:42.371878 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60582: BlockingRecv error: recv got EOF from 127.0.0.1:60582 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:06:42.372074 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 13239
I20260924 07:06:42.376819 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 13370
I20260924 07:06:42.381242 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 13501
I20260924 07:06:42.386042 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 13148
2026-09-24T07:06:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (869 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T07:06:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:06:42Z Disabled control of system clock
I20260924 07:06:42.412901 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:34063
--webserver_interface=127.3.217.62
--webserver_port=0
--builtin_ntp_servers=127.3.217.20:40299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:34063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 07:06:42.519745 13641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:42.520008 13641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:42.520052 13641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:42.524729 13641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:06:42.524796 13641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:42.524811 13641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:42.524827 13641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:06:42.524840 13641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:06:42.529676 13641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:40299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:34063
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:34063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:42.530625 13641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:42.531893 13641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:42.539633 13646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:42.539739 13649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:42.539752 13647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:42.539856 13641 server_base.cc:1034] running on GCE node
I20260924 07:06:42.540727 13641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:42.542093 13641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:42.543292 13641 hybrid_clock.cc:648] HybridClock initialized: now 1790233602543264 us; error 39 us; skew 500 ppm
I20260924 07:06:42.545485 13641 webserver.cc:460] Webserver started at http://127.3.217.62:33713/ using document root <none> and password file <none>
I20260924 07:06:42.545998 13641 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:42.546064 13641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:42.546219 13641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:42.548148 13641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "c4975bc847fd45619db53aa6a12d8f29"
format_stamp: "Formatted at 2026-09-24 07:06:42 on dist-test-slave-2p7j"
server_key: "cbdb6f0d1c909e39ca9ec8a3c08178fb"
server_key_iv: "c1a418076930f519b23b7c1e267e13e6"
server_key_version: "encryptionkey@0"
I20260924 07:06:42.548887 13641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "c4975bc847fd45619db53aa6a12d8f29"
format_stamp: "Formatted at 2026-09-24 07:06:42 on dist-test-slave-2p7j"
server_key: "cbdb6f0d1c909e39ca9ec8a3c08178fb"
server_key_iv: "c1a418076930f519b23b7c1e267e13e6"
server_key_version: "encryptionkey@0"
I20260924 07:06:42.553588 13641 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:06:42.556344 13655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:42.557713 13641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:06:42.557828 13641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "c4975bc847fd45619db53aa6a12d8f29"
format_stamp: "Formatted at 2026-09-24 07:06:42 on dist-test-slave-2p7j"
server_key: "cbdb6f0d1c909e39ca9ec8a3c08178fb"
server_key_iv: "c1a418076930f519b23b7c1e267e13e6"
server_key_version: "encryptionkey@0"
I20260924 07:06:42.557940 13641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:42.562292 13641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:42.562845 13641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:42.562996 13641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:42.569350 13641 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:34063
I20260924 07:06:42.569365 13708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:34063 every 8 connection(s)
I20260924 07:06:42.570204 13641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:06:42.573177 13709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:42.577867 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 13641
I20260924 07:06:42.577971 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:06:42.578214 3940 external_mini_cluster.cc:1428] Setting key e1f1452736bab413e0b4e289eaab52d1
I20260924 07:06:42.578387 13709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4975bc847fd45619db53aa6a12d8f29: Bootstrap starting.
I20260924 07:06:42.580856 13709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4975bc847fd45619db53aa6a12d8f29: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:42.581552 13709 log.cc:824] T 00000000000000000000000000000000 P c4975bc847fd45619db53aa6a12d8f29: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:42.583447 13709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4975bc847fd45619db53aa6a12d8f29: No bootstrap required, opened a new log
I20260924 07:06:42.583482 13712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:42.579774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40004 (local address 127.3.217.62:34063)
0924 07:06:42.580395 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:42.580410 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:42.580443 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:06:42.581089 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:42.581106 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:42.581422 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:42.581703 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:42.582381 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:42.582386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:42.582396 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:06:42.582405 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:42.582454 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:06:42.582469 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:42.582560 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:42.582650 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 07:06:42.582872 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":127,"threads_started":1}
I20260924 07:06:42.587141 13709 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4975bc847fd45619db53aa6a12d8f29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4975bc847fd45619db53aa6a12d8f29" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 34063 } }
I20260924 07:06:42.587334 13709 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4975bc847fd45619db53aa6a12d8f29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:42.587390 13709 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4975bc847fd45619db53aa6a12d8f29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4975bc847fd45619db53aa6a12d8f29, State: Initialized, Role: FOLLOWER
I20260924 07:06:42.587793 13709 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4975bc847fd45619db53aa6a12d8f29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4975bc847fd45619db53aa6a12d8f29" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 34063 } }
I20260924 07:06:42.587913 13709 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4975bc847fd45619db53aa6a12d8f29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:06:42.587975 13709 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4975bc847fd45619db53aa6a12d8f29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:06:42.588086 13709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4975bc847fd45619db53aa6a12d8f29 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:42.588937 13709 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4975bc847fd45619db53aa6a12d8f29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4975bc847fd45619db53aa6a12d8f29" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 34063 } }
I20260924 07:06:42.589210 13709 leader_election.cc:302] T 00000000000000000000000000000000 P c4975bc847fd45619db53aa6a12d8f29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4975bc847fd45619db53aa6a12d8f29; no voters:
I20260924 07:06:42.589424 13709 leader_election.cc:288] T 00000000000000000000000000000000 P c4975bc847fd45619db53aa6a12d8f29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:06:42.589658 13714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4975bc847fd45619db53aa6a12d8f29 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:42.590022 13714 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4975bc847fd45619db53aa6a12d8f29 [term 1 LEADER]: Becoming Leader. State: Replica: c4975bc847fd45619db53aa6a12d8f29, State: Running, Role: LEADER
I20260924 07:06:42.590368 13714 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4975bc847fd45619db53aa6a12d8f29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4975bc847fd45619db53aa6a12d8f29" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 34063 } }
I20260924 07:06:42.590418 13709 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4975bc847fd45619db53aa6a12d8f29 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:06:42.591686 13715 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4975bc847fd45619db53aa6a12d8f29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4975bc847fd45619db53aa6a12d8f29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4975bc847fd45619db53aa6a12d8f29" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 34063 } } }
I20260924 07:06:42.591800 13715 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4975bc847fd45619db53aa6a12d8f29 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:42.592125 13726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:06:42.592083 13716 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4975bc847fd45619db53aa6a12d8f29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4975bc847fd45619db53aa6a12d8f29. Latest consensus state: current_term: 1 leader_uuid: "c4975bc847fd45619db53aa6a12d8f29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4975bc847fd45619db53aa6a12d8f29" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 34063 } } }
I20260924 07:06:42.592176 13716 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4975bc847fd45619db53aa6a12d8f29 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:42.594911 13726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:06:42.599833 13726 catalog_manager.cc:1347] Generated new cluster ID: 51dc6e62d9294c02b258e44dcf85574b
I20260924 07:06:42.599886 13726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:06:42.618043 13726 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:06:42.618348 13726 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:06:42.629200 13726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4975bc847fd45619db53aa6a12d8f29: Generated new TSK 0
I20260924 07:06:42.629896 13726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:06:42.636562 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:34063
--builtin_ntp_servers=127.3.217.20:40299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 07:06:42.742930 13733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:42.743283 13733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:42.743330 13733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:42.749032 13733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:42.749111 13733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:42.749186 13733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:06:42.754158 13733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:40299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:34063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:42.755196 13733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:42.756542 13733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:42.765823 13738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:42.765827 13741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:42.765827 13733 server_base.cc:1034] running on GCE node
W20260924 07:06:42.765811 13739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:42.766395 13733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:42.766993 13733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:42.768154 13733 hybrid_clock.cc:648] HybridClock initialized: now 1790233602768130 us; error 37 us; skew 500 ppm
I20260924 07:06:42.769919 13733 webserver.cc:460] Webserver started at http://127.3.217.1:42531/ using document root <none> and password file <none>
I20260924 07:06:42.770699 13733 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:42.770772 13733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:42.770959 13733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:42.772917 13733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "266ae094b0854e9b92d48ecc50bc62ee"
format_stamp: "Formatted at 2026-09-24 07:06:42 on dist-test-slave-2p7j"
server_key: "4f6022734f0914223a1f0f1106639e5a"
server_key_iv: "e2df6e4c84db29138e58b4028b2a22d0"
server_key_version: "encryptionkey@0"
I20260924 07:06:42.773425 13733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "266ae094b0854e9b92d48ecc50bc62ee"
format_stamp: "Formatted at 2026-09-24 07:06:42 on dist-test-slave-2p7j"
server_key: "4f6022734f0914223a1f0f1106639e5a"
server_key_iv: "e2df6e4c84db29138e58b4028b2a22d0"
server_key_version: "encryptionkey@0"
I20260924 07:06:42.776813 13733 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:06:42.779217 13747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:42.780694 13733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:06:42.780803 13733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "266ae094b0854e9b92d48ecc50bc62ee"
format_stamp: "Formatted at 2026-09-24 07:06:42 on dist-test-slave-2p7j"
server_key: "4f6022734f0914223a1f0f1106639e5a"
server_key_iv: "e2df6e4c84db29138e58b4028b2a22d0"
server_key_version: "encryptionkey@0"
I20260924 07:06:42.780913 13733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:42.785209 13733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:42.785828 13733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:42.785990 13733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:42.786628 13733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:42.787462 13733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:42.787562 13733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:42.787632 13733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:42.787662 13733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:42.796098 13733 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:45037
I20260924 07:06:42.796128 13860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:45037 every 8 connection(s)
I20260924 07:06:42.796923 13733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:06:42.803237 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 13733
I20260924 07:06:42.803382 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:06:42.803763 13712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:42.799211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:40409 (local address 127.3.217.62:34063)
0924 07:06:42.799374 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:42.799378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:42.800625 (+ 1247us) server_negotiation.cc:407] Connection header received
0924 07:06:42.801897 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:42.801900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:42.801943 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:42.802017 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:42.803078 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:42.803081 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:42.803083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:06:42.803093 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:42.803110 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:06:42.803118 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:42.803195 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:42.803320 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 07:06:42.803364 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 07:06:42.803948 3940 external_mini_cluster.cc:1428] Setting key 654a085965233e081035253b2c49b470
I20260924 07:06:42.804674 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:42.799613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:34063 (local address 127.3.217.1:40409)
0924 07:06:42.800472 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:42.800514 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:42.801694 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:42.802178 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:42.802184 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:42.802266 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:42.802791 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:06:42.802792 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:42.802830 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:06:42.802894 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:42.803188 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:42.803196 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:42.803202 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:06:42.803350 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 07:06:42.803890 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":116,"threads_started":1}
I20260924 07:06:42.805904 13861 heartbeater.cc:347] Connected to a master server at 127.3.217.62:34063
I20260924 07:06:42.806047 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:34063
--builtin_ntp_servers=127.3.217.20:40299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 07:06:42.806130 13861 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:42.806540 13861 heartbeater.cc:511] Master 127.3.217.62:34063 requested a full tablet report, sending...
I20260924 07:06:42.808015 13672 ts_manager.cc:194] Registered new tserver with Master: 266ae094b0854e9b92d48ecc50bc62ee (127.3.217.1:45037)
W20260924 07:06:42.911410 13864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:42.911698 13864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:42.911739 13864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:42.915701 13864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:42.915750 13864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:42.915802 13864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:06:42.920331 13864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:40299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:34063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:42.921264 13864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:42.922940 13864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:42.932724 13869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:42.932698 13872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:42.932775 13870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:42.933131 13864 server_base.cc:1034] running on GCE node
I20260924 07:06:42.933599 13864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:42.934211 13864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:42.935381 13864 hybrid_clock.cc:648] HybridClock initialized: now 1790233602935357 us; error 37 us; skew 500 ppm
I20260924 07:06:42.937189 13864 webserver.cc:460] Webserver started at http://127.3.217.2:40953/ using document root <none> and password file <none>
I20260924 07:06:42.937714 13864 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:42.937779 13864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:42.937953 13864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:42.939930 13864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "8783eba9e64b4f29b45ef54823f34445"
format_stamp: "Formatted at 2026-09-24 07:06:42 on dist-test-slave-2p7j"
server_key: "17a675011609dcad95ecd6902393a869"
server_key_iv: "dd679cbdc6af3a5b5df531c4d053ad16"
server_key_version: "encryptionkey@0"
I20260924 07:06:42.940443 13864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "8783eba9e64b4f29b45ef54823f34445"
format_stamp: "Formatted at 2026-09-24 07:06:42 on dist-test-slave-2p7j"
server_key: "17a675011609dcad95ecd6902393a869"
server_key_iv: "dd679cbdc6af3a5b5df531c4d053ad16"
server_key_version: "encryptionkey@0"
I20260924 07:06:42.943848 13864 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:06:42.946350 13878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:42.947868 13864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:06:42.948007 13864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "8783eba9e64b4f29b45ef54823f34445"
format_stamp: "Formatted at 2026-09-24 07:06:42 on dist-test-slave-2p7j"
server_key: "17a675011609dcad95ecd6902393a869"
server_key_iv: "dd679cbdc6af3a5b5df531c4d053ad16"
server_key_version: "encryptionkey@0"
I20260924 07:06:42.948114 13864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:42.952607 13864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:42.953195 13864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:42.953368 13864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:42.953912 13864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:42.954818 13864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:42.954870 13864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:42.954924 13864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:42.954958 13864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:42.966061 13864 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:45725
I20260924 07:06:42.966140 13991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:45725 every 8 connection(s)
I20260924 07:06:42.966969 13864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:06:42.972441 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 13864
I20260924 07:06:42.972556 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:06:42.972879 3940 external_mini_cluster.cc:1428] Setting key 3d8c5f2b3c23f687bfc6fcba09b98243
I20260924 07:06:42.973028 13712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:42.968968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:60467 (local address 127.3.217.62:34063)
0924 07:06:42.969116 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:42.969121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:42.970151 (+ 1030us) server_negotiation.cc:407] Connection header received
0924 07:06:42.971362 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:42.971365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:42.971401 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:42.971480 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:42.972583 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:42.972587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:42.972590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:06:42.972603 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:42.972624 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:06:42.972634 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:42.972744 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:42.972850 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 07:06:42.972893 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:06:42.973798 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:42.969391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:34063 (local address 127.3.217.2:60467)
0924 07:06:42.970018 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:42.970054 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:42.971156 (+ 1102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:42.971693 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:42.971700 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:42.971781 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:42.972301 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:06:42.972303 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:42.972345 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:06:42.972411 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:42.972726 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:42.972733 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:42.972737 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:06:42.972860 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 07:06:42.973061 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":110,"threads_started":1}
I20260924 07:06:42.974838 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:34063
--builtin_ntp_servers=127.3.217.20:40299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 07:06:42.974854 13992 heartbeater.cc:347] Connected to a master server at 127.3.217.62:34063
I20260924 07:06:42.975075 13992 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:42.975539 13992 heartbeater.cc:511] Master 127.3.217.62:34063 requested a full tablet report, sending...
I20260924 07:06:42.976819 13672 ts_manager.cc:194] Registered new tserver with Master: 8783eba9e64b4f29b45ef54823f34445 (127.3.217.2:45725)
W20260924 07:06:43.082798 13995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:43.083037 13995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:43.083086 13995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:43.087085 13995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:43.087136 13995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:43.087209 13995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:06:43.091820 13995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:40299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:34063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:43.092837 13995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:43.094084 13995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:43.103466 14003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:43.103456 14000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:43.103456 14001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:43.103679 13995 server_base.cc:1034] running on GCE node
I20260924 07:06:43.104215 13995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:43.104789 13995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:43.105968 13995 hybrid_clock.cc:648] HybridClock initialized: now 1790233603105940 us; error 39 us; skew 500 ppm
I20260924 07:06:43.107800 13995 webserver.cc:460] Webserver started at http://127.3.217.3:38073/ using document root <none> and password file <none>
I20260924 07:06:43.108618 13995 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:43.108741 13995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:43.109054 13995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:43.110984 13995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "9556b8c98d0c4f16a231e5907d70bd1a"
format_stamp: "Formatted at 2026-09-24 07:06:43 on dist-test-slave-2p7j"
server_key: "bde5672b962ffe69839955a84007c717"
server_key_iv: "5bbbb2242ba7b1c95e6323bead0d0694"
server_key_version: "encryptionkey@0"
I20260924 07:06:43.111476 13995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "9556b8c98d0c4f16a231e5907d70bd1a"
format_stamp: "Formatted at 2026-09-24 07:06:43 on dist-test-slave-2p7j"
server_key: "bde5672b962ffe69839955a84007c717"
server_key_iv: "5bbbb2242ba7b1c95e6323bead0d0694"
server_key_version: "encryptionkey@0"
I20260924 07:06:43.115067 13995 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:06:43.117403 14009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:43.118538 13995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:06:43.118654 13995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "9556b8c98d0c4f16a231e5907d70bd1a"
format_stamp: "Formatted at 2026-09-24 07:06:43 on dist-test-slave-2p7j"
server_key: "bde5672b962ffe69839955a84007c717"
server_key_iv: "5bbbb2242ba7b1c95e6323bead0d0694"
server_key_version: "encryptionkey@0"
I20260924 07:06:43.118744 13995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:43.122938 13995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:43.123451 13995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:43.123620 13995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:43.124102 13995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:43.124905 13995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:43.124950 13995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:43.124997 13995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:43.125023 13995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:43.133137 13995 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:38065
I20260924 07:06:43.133164 14122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:38065 every 8 connection(s)
I20260924 07:06:43.133898 13995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:06:43.139931 13712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:43.136113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:47013 (local address 127.3.217.62:34063)
0924 07:06:43.136230 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:43.136234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:43.137239 (+ 1005us) server_negotiation.cc:407] Connection header received
0924 07:06:43.138472 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:43.138475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:43.138520 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:43.138592 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:43.139537 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:43.139540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:43.139543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:06:43.139554 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:43.139574 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:06:43.139582 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:43.139654 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:43.139767 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 07:06:43.139807 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:06:43.140657 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:43.136443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:34063 (local address 127.3.217.3:47013)
0924 07:06:43.137104 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:43.137140 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:43.138274 (+ 1134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:43.138737 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:43.138743 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:43.138820 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:43.139290 (+ 470us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:06:43.139291 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:43.139326 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:06:43.139386 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:43.139648 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:43.139656 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:43.139661 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:06:43.139800 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 07:06:43.140033 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":118,"threads_started":1}
I20260924 07:06:43.141141 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 13995
I20260924 07:06:43.141242 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:06:43.141527 3940 external_mini_cluster.cc:1428] Setting key 97cf4d01bc05d443a9b37f826a2ded3d
I20260924 07:06:43.141664 14123 heartbeater.cc:347] Connected to a master server at 127.3.217.62:34063
I20260924 07:06:43.141860 14123 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:43.142203 14123 heartbeater.cc:511] Master 127.3.217.62:34063 requested a full tablet report, sending...
I20260924 07:06:43.143101 13673 ts_manager.cc:194] Registered new tserver with Master: 9556b8c98d0c4f16a231e5907d70bd1a (127.3.217.3:38065)
I20260924 07:06:43.154748 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 07:06:43.157706 13712 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:06:43.156865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40012 (local address 127.3.217.62:34063)
0924 07:06:43.157064 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:43.157069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:43.157252 (+ 183us) server_negotiation.cc:407] Connection header received
0924 07:06:43.157316 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:43.157318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:43.157353 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:43.157417 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:43.157606 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40012: BlockingRecv error: recv got EOF from 127.0.0.1:40012 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 07:06:43.158335 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 13733
I20260924 07:06:43.163388 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 13864
I20260924 07:06:43.167867 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 13995
I20260924 07:06:43.172921 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 13641
2026-09-24T07:06:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (787 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:06:43 dist-test-slave-2p7j krb5kdc[14133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:06:43 dist-test-slave-2p7j krb5kdc[14133](info): set up 2 sockets
Sep 24 07:06:43 dist-test-slave-2p7j krb5kdc[14133](info): commencing operation
krb5kdc: starting...
W20260924 07:06:44.141361 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:06:44 dist-test-slave-2p7j krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:06:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:06:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:06:44.266896 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:33405
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:39975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:33405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:44.375339 14146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:44.375630 14146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:44.375679 14146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:44.379853 14146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:06:44.379909 14146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:44.379928 14146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:44.379944 14146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:06:44.379963 14146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:06:44.386102 14146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:39975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:33405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:33405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:44.387042 14146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:44.387987 14146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:44.395800 14154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:44.395859 14151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:44.395808 14152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:44.395968 14146 server_base.cc:1034] running on GCE node
I20260924 07:06:44.396539 14146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:44.397519 14146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:44.398689 14146 hybrid_clock.cc:648] HybridClock initialized: now 1790233604398618 us; error 80 us; skew 500 ppm
Sep 24 07:06:44 dist-test-slave-2p7j krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233604, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:44.401321 14146 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:06:44.402097 14146 webserver.cc:460] Webserver started at http://127.3.217.62:45459/ using document root <none> and password file <none>
I20260924 07:06:44.402578 14146 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:44.402635 14146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:44.402779 14146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:44.404603 14146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "ac7b91b3f8884fc2a1c66e69be040866"
format_stamp: "Formatted at 2026-09-24 07:06:44 on dist-test-slave-2p7j"
server_key: "bd48149a2d654eeccbc01828b16cbb68"
server_key_iv: "a3360094bf463e99b3bf83137c70c8d8"
server_key_version: "encryptionkey@0"
I20260924 07:06:44.405072 14146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "ac7b91b3f8884fc2a1c66e69be040866"
format_stamp: "Formatted at 2026-09-24 07:06:44 on dist-test-slave-2p7j"
server_key: "bd48149a2d654eeccbc01828b16cbb68"
server_key_iv: "a3360094bf463e99b3bf83137c70c8d8"
server_key_version: "encryptionkey@0"
I20260924 07:06:44.408365 14146 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:06:44.410495 14161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:44.411676 14146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:06:44.411773 14146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "ac7b91b3f8884fc2a1c66e69be040866"
format_stamp: "Formatted at 2026-09-24 07:06:44 on dist-test-slave-2p7j"
server_key: "bd48149a2d654eeccbc01828b16cbb68"
server_key_iv: "a3360094bf463e99b3bf83137c70c8d8"
server_key_version: "encryptionkey@0"
I20260924 07:06:44.411849 14146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:44.431495 14146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:44.434219 14146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:44.434347 14146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:44.440223 14146 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:33405
I20260924 07:06:44.440440 14213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:33405 every 8 connection(s)
I20260924 07:06:44.441025 14146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:06:44.442636 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 14146
I20260924 07:06:44.442730 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:06:44.442963 3940 external_mini_cluster.cc:1428] Setting key 97623eb0074f64c6e1ea32029b469142
I20260924 07:06:44.443445 14214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:44.448675 14214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac7b91b3f8884fc2a1c66e69be040866: Bootstrap starting.
Sep 24 07:06:44 dist-test-slave-2p7j krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:06:44.450616 14214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac7b91b3f8884fc2a1c66e69be040866: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:44.451396 14214 log.cc:824] T 00000000000000000000000000000000 P ac7b91b3f8884fc2a1c66e69be040866: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:44.453267 14214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac7b91b3f8884fc2a1c66e69be040866: No bootstrap required, opened a new log
I20260924 07:06:44.455173 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:44.444176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57074 (local address 127.3.217.62:33405)
0924 07:06:44.444556 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:44.444569 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:44.444609 (+ 40us) server_negotiation.cc:407] Connection header received
0924 07:06:44.445205 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:44.445219 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:44.445491 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:44.445770 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:44.446459 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:44.447390 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:44.448180 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:44.448613 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:44.448755 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:44.450799 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:44.450806 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:44.450821 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:44.450851 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:44.452503 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:44.452875 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:44.452881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:44.452886 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:44.453010 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:44.453234 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:44.453237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:44.453239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:44.453693 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:44.453864 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:44.454065 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 07:06:44.454323 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":124,"threads_started":1}
I20260924 07:06:44.457309 14214 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac7b91b3f8884fc2a1c66e69be040866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7b91b3f8884fc2a1c66e69be040866" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33405 } }
I20260924 07:06:44.457518 14214 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac7b91b3f8884fc2a1c66e69be040866 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:44.457559 14214 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac7b91b3f8884fc2a1c66e69be040866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac7b91b3f8884fc2a1c66e69be040866, State: Initialized, Role: FOLLOWER
I20260924 07:06:44.457885 14214 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac7b91b3f8884fc2a1c66e69be040866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7b91b3f8884fc2a1c66e69be040866" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33405 } }
I20260924 07:06:44.457985 14214 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac7b91b3f8884fc2a1c66e69be040866 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:06:44.458022 14214 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac7b91b3f8884fc2a1c66e69be040866 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:06:44.458078 14214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac7b91b3f8884fc2a1c66e69be040866 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:44.458818 14214 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac7b91b3f8884fc2a1c66e69be040866 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7b91b3f8884fc2a1c66e69be040866" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33405 } }
I20260924 07:06:44.459079 14214 leader_election.cc:302] T 00000000000000000000000000000000 P ac7b91b3f8884fc2a1c66e69be040866 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac7b91b3f8884fc2a1c66e69be040866; no voters:
I20260924 07:06:44.459290 14214 leader_election.cc:288] T 00000000000000000000000000000000 P ac7b91b3f8884fc2a1c66e69be040866 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:06:44.459380 14219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac7b91b3f8884fc2a1c66e69be040866 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:44.459616 14219 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac7b91b3f8884fc2a1c66e69be040866 [term 1 LEADER]: Becoming Leader. State: Replica: ac7b91b3f8884fc2a1c66e69be040866, State: Running, Role: LEADER
I20260924 07:06:44.459926 14219 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac7b91b3f8884fc2a1c66e69be040866 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7b91b3f8884fc2a1c66e69be040866" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33405 } }
I20260924 07:06:44.460355 14214 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac7b91b3f8884fc2a1c66e69be040866 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:06:44.461395 14221 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac7b91b3f8884fc2a1c66e69be040866 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac7b91b3f8884fc2a1c66e69be040866. Latest consensus state: current_term: 1 leader_uuid: "ac7b91b3f8884fc2a1c66e69be040866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7b91b3f8884fc2a1c66e69be040866" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33405 } } }
I20260924 07:06:44.461618 14221 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac7b91b3f8884fc2a1c66e69be040866 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:44.461395 14220 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac7b91b3f8884fc2a1c66e69be040866 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac7b91b3f8884fc2a1c66e69be040866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7b91b3f8884fc2a1c66e69be040866" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33405 } } }
I20260924 07:06:44.461877 14220 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac7b91b3f8884fc2a1c66e69be040866 [sys.catalog]: This master's current role is: LEADER
I20260924 07:06:44.462644 14231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:06:44.465159 14231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:06:44.470475 14231 catalog_manager.cc:1347] Generated new cluster ID: ae8c974cb4e24bb49fad5d2b5ad54b26
I20260924 07:06:44.470532 14231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:06:44.484303 14231 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:06:44.485282 14231 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:06:44.492756 14231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac7b91b3f8884fc2a1c66e69be040866: Generated new TSK 0
I20260924 07:06:44.493373 14231 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
I20260924 07:06:44.521824 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:33405
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:39975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:44.633880 14240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:44.634452 14240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:44.634575 14240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:44.638986 14240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:44.639063 14240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:44.639127 14240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:06:44.645278 14240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:39975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:33405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:44.646541 14240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:44.647496 14240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:44.656602 14246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:44.656664 14240 server_base.cc:1034] running on GCE node
W20260924 07:06:44.656605 14245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:44.656875 14248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:44.657399 14240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:44.658188 14240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:44.659390 14240 hybrid_clock.cc:648] HybridClock initialized: now 1790233604659343 us; error 61 us; skew 500 ppm
Sep 24 07:06:44 dist-test-slave-2p7j krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233604, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:44.663216 14240 init.cc:377] Logged in from keytab as kudu/127.3.217.1@KRBTEST.COM (short username kudu)
I20260924 07:06:44.664302 14240 webserver.cc:460] Webserver started at http://127.3.217.1:34361/ using document root <none> and password file <none>
I20260924 07:06:44.664804 14240 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:44.664862 14240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:44.665012 14240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:44.666886 14240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "e3e687a9a16048658a8ecaf90f3c15e3"
format_stamp: "Formatted at 2026-09-24 07:06:44 on dist-test-slave-2p7j"
server_key: "701c58bb5f041091cd364c8263f6978e"
server_key_iv: "d0555b03db9180f627434ab0cd4ab4ee"
server_key_version: "encryptionkey@0"
I20260924 07:06:44.667361 14240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "e3e687a9a16048658a8ecaf90f3c15e3"
format_stamp: "Formatted at 2026-09-24 07:06:44 on dist-test-slave-2p7j"
server_key: "701c58bb5f041091cd364c8263f6978e"
server_key_iv: "d0555b03db9180f627434ab0cd4ab4ee"
server_key_version: "encryptionkey@0"
I20260924 07:06:44.670936 14240 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260924 07:06:44.673755 14255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:44.675282 14240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 07:06:44.675483 14240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "e3e687a9a16048658a8ecaf90f3c15e3"
format_stamp: "Formatted at 2026-09-24 07:06:44 on dist-test-slave-2p7j"
server_key: "701c58bb5f041091cd364c8263f6978e"
server_key_iv: "d0555b03db9180f627434ab0cd4ab4ee"
server_key_version: "encryptionkey@0"
I20260924 07:06:44.675781 14240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:44.691854 14240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:44.694818 14240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:44.694976 14240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:44.695552 14240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:44.696305 14240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:44.696349 14240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:44.696388 14240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:44.696417 14240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:44.704938 14240 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:35489
I20260924 07:06:44.705086 14369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:35489 every 8 connection(s)
I20260924 07:06:44.705905 14240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:06:44.707799 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 14240
I20260924 07:06:44.707913 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:06:44.708228 3940 external_mini_cluster.cc:1428] Setting key 5a367291752e3abbe71c66a849dcbda4
Sep 24 07:06:44 dist-test-slave-2p7j krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233604, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.2@KRBTEST.COM" created.
I20260924 07:06:44.721225 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:44.707683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:40641 (local address 127.3.217.62:33405)
0924 07:06:44.709100 (+ 1417us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:44.709104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:44.709119 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:06:44.710266 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:44.710269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:44.710320 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:44.710389 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:44.711816 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:44.712357 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:44.713944 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:44.714187 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:44.714283 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:44.716899 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:44.716902 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:44.716904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:44.716927 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:44.718756 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:44.719595 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:44.719600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:44.719602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:44.719703 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:44.720071 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:44.720076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:44.720077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:44.720271 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:44.720366 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:44.720967 (+ 601us) server_negotiation.cc:299] Negotiation successful
0924 07:06:44.721080 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1334}
I20260924 07:06:44.722153 14372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:44.708155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:33405 (local address 127.3.217.1:40641)
0924 07:06:44.708836 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:44.708886 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:44.710075 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:44.710560 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:44.710566 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:44.711009 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:44.711662 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:44.711681 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:44.712476 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:44.712479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:44.713782 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:44.713794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:44.714341 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:44.714345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:44.714551 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:44.715150 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:44.715169 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:44.716750 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:44.718937 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:44.718951 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:44.718971 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:44.719417 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:44.719800 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:44.719804 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:44.719807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:44.719962 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:44.720376 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:44.720383 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:44.720699 (+ 316us) client_negotiation.cc:769] Sending connection context
0924 07:06:44.720894 (+ 195us) client_negotiation.cc:240] Negotiation successful
0924 07:06:44.721222 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":136,"threads_started":1}
I20260924 07:06:44.723265 14370 heartbeater.cc:347] Connected to a master server at 127.3.217.62:33405
I20260924 07:06:44.723717 14370 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:44.724471 14370 heartbeater.cc:511] Master 127.3.217.62:33405 requested a full tablet report, sending...
I20260924 07:06:44.726794 14178 ts_manager.cc:194] Registered new tserver with Master: e3e687a9a16048658a8ecaf90f3c15e3 (127.3.217.1:35489)
I20260924 07:06:44.728106 14178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.1@KRBTEST.COM'} at 127.3.217.1:40641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
I20260924 07:06:44.738283 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:33405
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:39975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:44.851994 14375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:44.852398 14375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:44.852519 14375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:44.857411 14375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:44.857483 14375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:44.857555 14375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:06:44.862825 14375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:39975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:33405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:44.863765 14375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:44.864683 14375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:06:44.874271 14375 server_base.cc:1034] running on GCE node
W20260924 07:06:44.874130 14381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:44.874049 14380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:44.874130 14383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:44.874887 14375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:44.875476 14375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:44.876683 14375 hybrid_clock.cc:648] HybridClock initialized: now 1790233604876665 us; error 32 us; skew 500 ppm
Sep 24 07:06:44 dist-test-slave-2p7j krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233604, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:44.879227 14375 init.cc:377] Logged in from keytab as kudu/127.3.217.2@KRBTEST.COM (short username kudu)
I20260924 07:06:44.880040 14375 webserver.cc:460] Webserver started at http://127.3.217.2:41315/ using document root <none> and password file <none>
I20260924 07:06:44.880542 14375 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:44.880609 14375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:44.880803 14375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:44.882628 14375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "1b9c5c1547b34195976b56a3a591d875"
format_stamp: "Formatted at 2026-09-24 07:06:44 on dist-test-slave-2p7j"
server_key: "fba238affb30cc3342e9e010d762375a"
server_key_iv: "b97905a872bd8ec7a3bd31bc79136ddb"
server_key_version: "encryptionkey@0"
I20260924 07:06:44.883122 14375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "1b9c5c1547b34195976b56a3a591d875"
format_stamp: "Formatted at 2026-09-24 07:06:44 on dist-test-slave-2p7j"
server_key: "fba238affb30cc3342e9e010d762375a"
server_key_iv: "b97905a872bd8ec7a3bd31bc79136ddb"
server_key_version: "encryptionkey@0"
I20260924 07:06:44.886479 14375 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 07:06:44.888718 14390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:44.889855 14375 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:06:44.889954 14375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "1b9c5c1547b34195976b56a3a591d875"
format_stamp: "Formatted at 2026-09-24 07:06:44 on dist-test-slave-2p7j"
server_key: "fba238affb30cc3342e9e010d762375a"
server_key_iv: "b97905a872bd8ec7a3bd31bc79136ddb"
server_key_version: "encryptionkey@0"
I20260924 07:06:44.890039 14375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:44.902242 14375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:44.904995 14375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:44.905164 14375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:44.905669 14375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:44.906422 14375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:44.906476 14375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:44.906527 14375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:44.906550 14375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:44.914458 14375 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:46201
I20260924 07:06:44.914480 14504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:46201 every 8 connection(s)
I20260924 07:06:44.915324 14375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
Sep 24 07:06:44 dist-test-slave-2p7j krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233604, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:06:44.925277 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 14375
I20260924 07:06:44.925390 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:06:44.925670 3940 external_mini_cluster.cc:1428] Setting key d1881285d11ae61968c3ca3afd481d70
I20260924 07:06:44.928126 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:44.917254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:59409 (local address 127.3.217.62:33405)
0924 07:06:44.917417 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:44.917421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:44.918238 (+ 817us) server_negotiation.cc:407] Connection header received
0924 07:06:44.919113 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:44.919117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:44.919179 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:44.919261 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:44.920405 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:44.920938 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:44.922037 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:44.922262 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:44.922352 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:44.924750 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:44.924753 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:44.924756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:44.924778 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:44.926193 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:44.926681 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:44.926685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:44.926687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:44.926769 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:44.927115 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:44.927118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:44.927120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:44.927293 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:44.927376 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:44.927858 (+ 482us) server_negotiation.cc:299] Negotiation successful
0924 07:06:44.927974 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 07:06:44.928812 14507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:44.917402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:33405 (local address 127.3.217.2:59409)
0924 07:06:44.918061 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:44.918102 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:44.918890 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:44.919435 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:44.919442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:44.919807 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:44.920251 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:44.920264 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:44.921064 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:44.921067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:44.921909 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:44.921919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:44.922417 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:44.922422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:44.922576 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:44.923091 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:44.923107 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:44.924590 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:44.926282 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:44.926287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:44.926307 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:44.926575 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:44.926851 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:44.926853 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:44.926855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:44.927031 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:44.927385 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:44.927391 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:44.927647 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 07:06:44.927804 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 07:06:44.928096 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":117,"threads_started":1}
I20260924 07:06:44.930289 14505 heartbeater.cc:347] Connected to a master server at 127.3.217.62:33405
I20260924 07:06:44.930569 14505 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:44.931061 14505 heartbeater.cc:511] Master 127.3.217.62:33405 requested a full tablet report, sending...
I20260924 07:06:44.932071 14178 ts_manager.cc:194] Registered new tserver with Master: 1b9c5c1547b34195976b56a3a591d875 (127.3.217.2:46201)
I20260924 07:06:44.932622 14178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.2@KRBTEST.COM'} at 127.3.217.2:59409
WARNING: no policy specified for kudu/127.3.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
I20260924 07:06:44.955231 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:33405
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:39975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:06:45.066483 14510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:06:45.066748 14510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:06:45.066793 14510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:06:45.071051 14510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:06:45.071112 14510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:06:45.071173 14510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:06:45.078722 14510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:39975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:33405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:06:45.079767 14510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:06:45.080758 14510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:06:45.091856 14518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:45.091938 14516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:06:45.091885 14515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:06:45.092118 14510 server_base.cc:1034] running on GCE node
I20260924 07:06:45.092504 14510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:06:45.093102 14510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:06:45.094274 14510 hybrid_clock.cc:648] HybridClock initialized: now 1790233605094258 us; error 27 us; skew 500 ppm
Sep 24 07:06:45 dist-test-slave-2p7j krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233605, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:06:45.096902 14510 init.cc:377] Logged in from keytab as kudu/127.3.217.3@KRBTEST.COM (short username kudu)
I20260924 07:06:45.097723 14510 webserver.cc:460] Webserver started at http://127.3.217.3:40125/ using document root <none> and password file <none>
I20260924 07:06:45.098225 14510 fs_manager.cc:359] Metadata directory not provided
I20260924 07:06:45.098287 14510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:06:45.098459 14510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:06:45.100854 14510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "f7094b3bdb804789b8ba9d2b43f78f5d"
format_stamp: "Formatted at 2026-09-24 07:06:45 on dist-test-slave-2p7j"
server_key: "41128cea3e120b8e6682c7fc8c9c2420"
server_key_iv: "f95aaee45654b8a22fd0addfefe1ddc4"
server_key_version: "encryptionkey@0"
I20260924 07:06:45.101949 14510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "f7094b3bdb804789b8ba9d2b43f78f5d"
format_stamp: "Formatted at 2026-09-24 07:06:45 on dist-test-slave-2p7j"
server_key: "41128cea3e120b8e6682c7fc8c9c2420"
server_key_iv: "f95aaee45654b8a22fd0addfefe1ddc4"
server_key_version: "encryptionkey@0"
I20260924 07:06:45.106513 14510 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:06:45.109586 14525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:45.111153 14510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:06:45.111266 14510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "f7094b3bdb804789b8ba9d2b43f78f5d"
format_stamp: "Formatted at 2026-09-24 07:06:45 on dist-test-slave-2p7j"
server_key: "41128cea3e120b8e6682c7fc8c9c2420"
server_key_iv: "f95aaee45654b8a22fd0addfefe1ddc4"
server_key_version: "encryptionkey@0"
I20260924 07:06:45.111357 14510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:06:45.130826 14510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:06:45.133422 14510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:06:45.133577 14510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:06:45.134083 14510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:06:45.134811 14510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:06:45.134855 14510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:45.134903 14510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:06:45.134930 14510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:06:45.144424 14510 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:39493
I20260924 07:06:45.144471 14638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:39493 every 8 connection(s)
I20260924 07:06:45.145221 14510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:06:45.153301 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 14510
I20260924 07:06:45.153434 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:06:45.153855 3940 external_mini_cluster.cc:1428] Setting key 6b38a6c0143821a44ca8edd6a6b60e0a
Sep 24 07:06:45 dist-test-slave-2p7j krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233605, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:06:45.158341 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:45.146838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:43661 (local address 127.3.217.62:33405)
0924 07:06:45.146959 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:45.146963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:45.147771 (+ 808us) server_negotiation.cc:407] Connection header received
0924 07:06:45.149059 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:45.149062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:45.149130 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:45.149202 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:45.150267 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.150779 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.152039 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.152258 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.152344 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:45.155280 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:45.155283 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:45.155286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:45.155312 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:45.156720 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:45.157189 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:45.157192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:45.157193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:45.157265 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:45.157527 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:45.157529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:45.157530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:45.157679 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:45.157745 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:45.158142 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 07:06:45.158232 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:06:45.159018 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:45.147166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:33405 (local address 127.3.217.3:43661)
0924 07:06:45.147646 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:45.147684 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:45.148870 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:45.149365 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:45.149371 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:45.149707 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:06:45.150140 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:45.150152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.150887 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.150890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:45.151920 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:45.151931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.152384 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.152387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:45.152532 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:45.153115 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:06:45.153138 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:06:45.155112 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:06:45.156820 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:45.156825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:45.156835 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:45.157098 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:45.157332 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:06:45.157334 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:06:45.157335 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:06:45.157449 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:06:45.157744 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:06:45.157748 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:06:45.157942 (+ 194us) client_negotiation.cc:769] Sending connection context
0924 07:06:45.158086 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 07:06:45.158306 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":104,"threads_started":1}
I20260924 07:06:45.160034 14639 heartbeater.cc:347] Connected to a master server at 127.3.217.62:33405
I20260924 07:06:45.160465 14639 heartbeater.cc:464] Registering TS with master...
I20260924 07:06:45.161041 14639 heartbeater.cc:511] Master 127.3.217.62:33405 requested a full tablet report, sending...
I20260924 07:06:45.162063 14178 ts_manager.cc:194] Registered new tserver with Master: f7094b3bdb804789b8ba9d2b43f78f5d (127.3.217.3:39493)
I20260924 07:06:45.162640 14178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.3@KRBTEST.COM'} at 127.3.217.3:43661
I20260924 07:06:45.167632 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:06:45.176142 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:45.169472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57084 (local address 127.3.217.62:33405)
0924 07:06:45.169586 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:45.169590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:45.169949 (+ 359us) server_negotiation.cc:407] Connection header received
0924 07:06:45.170122 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:45.170124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:45.170160 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:45.170238 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:06:45.170926 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.171434 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.172276 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.172479 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.172555 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:45.173735 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:06:45.173738 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:06:45.173740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:06:45.173760 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:06:45.174867 (+ 1107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:45.175215 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:45.175218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:45.175219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:45.175283 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:06:45.175456 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:06:45.175458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:06:45.175459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:06:45.175680 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:06:45.175779 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:45.175946 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 07:06:45.176029 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 07:06:45.178597 14178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:06:45.181558 14178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:06:45.196412 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:45.191200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:35489 (local address 127.0.0.1:39560)
0924 07:06:45.192176 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:45.192187 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:45.192271 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:45.192663 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:45.192666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:45.192684 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:45.192800 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:45.192807 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.194032 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.194036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:45.195123 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:45.195131 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.196079 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.196082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:45.196171 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:45.196190 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 07:06:45.196245 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 07:06:45.196295 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"mutex_wait_us":75,"thread_start_us":83,"threads_started":1}
I20260924 07:06:45.196820 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:45.191618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39560 (local address 127.3.217.1:35489)
0924 07:06:45.191931 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:45.191945 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:45.192268 (+ 323us) server_negotiation.cc:407] Connection header received
0924 07:06:45.192366 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:45.192370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:45.192529 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:45.192666 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:45.192907 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.193897 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.195224 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.195801 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.195892 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:45.196024 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:45.196602 (+ 578us) server_negotiation.cc:299] Negotiation successful
0924 07:06:45.196693 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":65,"threads_started":1}
I20260924 07:06:45.196412 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:45.190514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:46201 (local address 127.0.0.1:46864)
0924 07:06:45.191140 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:45.191188 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:45.191310 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:45.192070 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:45.192074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:45.192090 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:45.192185 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:45.192192 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.193285 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.193289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:45.194765 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:45.194774 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.195903 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.195907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:45.196010 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:45.196047 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 07:06:45.196144 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 07:06:45.196253 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":19,"thread_start_us":91,"threads_started":1}
I20260924 07:06:45.196938 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:45.190662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45558 (local address 127.3.217.3:39493)
0924 07:06:45.191449 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:45.191460 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:45.191478 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:06:45.191580 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:45.191586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:45.191777 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:45.192231 (+ 454us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:45.192365 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.193517 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.195255 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.196033 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.196128 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:45.196251 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:45.196611 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 07:06:45.196768 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":111,"threads_started":1}
I20260924 07:06:45.196916 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:45.190523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:39493 (local address 127.0.0.1:45558)
0924 07:06:45.191275 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:45.191288 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:45.191375 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:45.191942 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:45.191948 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:45.191971 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:45.192097 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:45.192104 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.193690 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.193694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:45.195115 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:45.195125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.196129 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.196132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:45.196234 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:45.196256 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:06:45.196558 (+ 302us) client_negotiation.cc:240] Negotiation successful
0924 07:06:45.196701 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":180,"thread_start_us":97,"threads_started":1}
I20260924 07:06:45.197378 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:45.190830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46864 (local address 127.3.217.2:46201)
0924 07:06:45.191236 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:45.191247 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:45.191272 (+ 25us) server_negotiation.cc:407] Connection header received
0924 07:06:45.191543 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:45.191550 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:45.191760 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:45.191907 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:45.192304 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.193180 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.194885 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.195778 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.195889 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:45.196040 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:45.196480 (+ 440us) server_negotiation.cc:299] Negotiation successful
0924 07:06:45.196776 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":130,"threads_started":1}
I20260924 07:06:45.199765 14573 tablet_service.cc:1467] Processing CreateTablet for tablet 63a403dfa59e415ea4ff702bb767fb94 (DEFAULT_TABLE table=test-table [id=dcb5fe4b80564e9391bee21c1101919c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:45.200655 14573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a403dfa59e415ea4ff702bb767fb94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:45.200598 14439 tablet_service.cc:1467] Processing CreateTablet for tablet 63a403dfa59e415ea4ff702bb767fb94 (DEFAULT_TABLE table=test-table [id=dcb5fe4b80564e9391bee21c1101919c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:45.200592 14303 tablet_service.cc:1467] Processing CreateTablet for tablet 63a403dfa59e415ea4ff702bb767fb94 (DEFAULT_TABLE table=test-table [id=dcb5fe4b80564e9391bee21c1101919c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:06:45.201537 14439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a403dfa59e415ea4ff702bb767fb94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:45.201548 14303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a403dfa59e415ea4ff702bb767fb94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:06:45.206934 14656 tablet_bootstrap.cc:492] T 63a403dfa59e415ea4ff702bb767fb94 P f7094b3bdb804789b8ba9d2b43f78f5d: Bootstrap starting.
I20260924 07:06:45.208951 14657 tablet_bootstrap.cc:492] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875: Bootstrap starting.
I20260924 07:06:45.209328 14656 tablet_bootstrap.cc:654] T 63a403dfa59e415ea4ff702bb767fb94 P f7094b3bdb804789b8ba9d2b43f78f5d: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:45.209439 14658 tablet_bootstrap.cc:492] T 63a403dfa59e415ea4ff702bb767fb94 P e3e687a9a16048658a8ecaf90f3c15e3: Bootstrap starting.
I20260924 07:06:45.210163 14656 log.cc:824] T 63a403dfa59e415ea4ff702bb767fb94 P f7094b3bdb804789b8ba9d2b43f78f5d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:45.211081 14658 tablet_bootstrap.cc:654] T 63a403dfa59e415ea4ff702bb767fb94 P e3e687a9a16048658a8ecaf90f3c15e3: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:45.211184 14657 tablet_bootstrap.cc:654] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875: Neither blocks nor log segments found. Creating new log.
I20260924 07:06:45.211977 14657 log.cc:824] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:45.212203 14658 log.cc:824] T 63a403dfa59e415ea4ff702bb767fb94 P e3e687a9a16048658a8ecaf90f3c15e3: Log is configured to *not* fsync() on all Append() calls
I20260924 07:06:45.212514 14656 tablet_bootstrap.cc:492] T 63a403dfa59e415ea4ff702bb767fb94 P f7094b3bdb804789b8ba9d2b43f78f5d: No bootstrap required, opened a new log
I20260924 07:06:45.212678 14656 ts_tablet_manager.cc:1397] T 63a403dfa59e415ea4ff702bb767fb94 P f7094b3bdb804789b8ba9d2b43f78f5d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260924 07:06:45.213907 14658 tablet_bootstrap.cc:492] T 63a403dfa59e415ea4ff702bb767fb94 P e3e687a9a16048658a8ecaf90f3c15e3: No bootstrap required, opened a new log
I20260924 07:06:45.214097 14658 ts_tablet_manager.cc:1397] T 63a403dfa59e415ea4ff702bb767fb94 P e3e687a9a16048658a8ecaf90f3c15e3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 07:06:45.215472 14656 raft_consensus.cc:348] T 63a403dfa59e415ea4ff702bb767fb94 P f7094b3bdb804789b8ba9d2b43f78f5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7094b3bdb804789b8ba9d2b43f78f5d" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 39493 } } peers { permanent_uuid: "1b9c5c1547b34195976b56a3a591d875" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 46201 } } peers { permanent_uuid: "e3e687a9a16048658a8ecaf90f3c15e3" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35489 } }
I20260924 07:06:45.216032 14656 raft_consensus.cc:374] T 63a403dfa59e415ea4ff702bb767fb94 P f7094b3bdb804789b8ba9d2b43f78f5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:45.216112 14656 raft_consensus.cc:729] T 63a403dfa59e415ea4ff702bb767fb94 P f7094b3bdb804789b8ba9d2b43f78f5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7094b3bdb804789b8ba9d2b43f78f5d, State: Initialized, Role: FOLLOWER
I20260924 07:06:45.216635 14656 consensus_queue.cc:260] T 63a403dfa59e415ea4ff702bb767fb94 P f7094b3bdb804789b8ba9d2b43f78f5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7094b3bdb804789b8ba9d2b43f78f5d" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 39493 } } peers { permanent_uuid: "1b9c5c1547b34195976b56a3a591d875" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 46201 } } peers { permanent_uuid: "e3e687a9a16048658a8ecaf90f3c15e3" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35489 } }
I20260924 07:06:45.216997 14658 raft_consensus.cc:348] T 63a403dfa59e415ea4ff702bb767fb94 P e3e687a9a16048658a8ecaf90f3c15e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7094b3bdb804789b8ba9d2b43f78f5d" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 39493 } } peers { permanent_uuid: "1b9c5c1547b34195976b56a3a591d875" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 46201 } } peers { permanent_uuid: "e3e687a9a16048658a8ecaf90f3c15e3" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35489 } }
I20260924 07:06:45.217211 14658 raft_consensus.cc:374] T 63a403dfa59e415ea4ff702bb767fb94 P e3e687a9a16048658a8ecaf90f3c15e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:45.217267 14658 raft_consensus.cc:729] T 63a403dfa59e415ea4ff702bb767fb94 P e3e687a9a16048658a8ecaf90f3c15e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3e687a9a16048658a8ecaf90f3c15e3, State: Initialized, Role: FOLLOWER
I20260924 07:06:45.217433 14639 heartbeater.cc:503] Master 127.3.217.62:33405 was elected leader, sending a full tablet report...
I20260924 07:06:45.217731 14656 ts_tablet_manager.cc:1428] T 63a403dfa59e415ea4ff702bb767fb94 P f7094b3bdb804789b8ba9d2b43f78f5d: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260924 07:06:45.218638 14658 consensus_queue.cc:260] T 63a403dfa59e415ea4ff702bb767fb94 P e3e687a9a16048658a8ecaf90f3c15e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7094b3bdb804789b8ba9d2b43f78f5d" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 39493 } } peers { permanent_uuid: "1b9c5c1547b34195976b56a3a591d875" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 46201 } } peers { permanent_uuid: "e3e687a9a16048658a8ecaf90f3c15e3" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35489 } }
I20260924 07:06:45.219276 14657 tablet_bootstrap.cc:492] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875: No bootstrap required, opened a new log
I20260924 07:06:45.219448 14657 ts_tablet_manager.cc:1397] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260924 07:06:45.219596 14658 ts_tablet_manager.cc:1428] T 63a403dfa59e415ea4ff702bb767fb94 P e3e687a9a16048658a8ecaf90f3c15e3: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260924 07:06:45.219887 14370 heartbeater.cc:503] Master 127.3.217.62:33405 was elected leader, sending a full tablet report...
I20260924 07:06:45.222685 14657 raft_consensus.cc:348] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7094b3bdb804789b8ba9d2b43f78f5d" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 39493 } } peers { permanent_uuid: "1b9c5c1547b34195976b56a3a591d875" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 46201 } } peers { permanent_uuid: "e3e687a9a16048658a8ecaf90f3c15e3" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35489 } }
I20260924 07:06:45.222860 14657 raft_consensus.cc:374] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:06:45.222896 14657 raft_consensus.cc:729] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b9c5c1547b34195976b56a3a591d875, State: Initialized, Role: FOLLOWER
I20260924 07:06:45.223191 14657 consensus_queue.cc:260] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7094b3bdb804789b8ba9d2b43f78f5d" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 39493 } } peers { permanent_uuid: "1b9c5c1547b34195976b56a3a591d875" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 46201 } } peers { permanent_uuid: "e3e687a9a16048658a8ecaf90f3c15e3" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35489 } }
I20260924 07:06:45.223721 14505 heartbeater.cc:503] Master 127.3.217.62:33405 was elected leader, sending a full tablet report...
I20260924 07:06:45.223881 14657 ts_tablet_manager.cc:1428] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260924 07:06:45.396090 14640 tablet.cc:2369] T 63a403dfa59e415ea4ff702bb767fb94 P f7094b3bdb804789b8ba9d2b43f78f5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:06:45.398108 14664 raft_consensus.cc:482] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:06:45.398276 14664 raft_consensus.cc:504] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7094b3bdb804789b8ba9d2b43f78f5d" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 39493 } } peers { permanent_uuid: "1b9c5c1547b34195976b56a3a591d875" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 46201 } } peers { permanent_uuid: "e3e687a9a16048658a8ecaf90f3c15e3" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35489 } }
I20260924 07:06:45.399351 14664 leader_election.cc:288] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7094b3bdb804789b8ba9d2b43f78f5d (127.3.217.3:39493), e3e687a9a16048658a8ecaf90f3c15e3 (127.3.217.1:35489)
I20260924 07:06:45.404148 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:45.400037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:59515 (local address 127.3.217.1:35489)
0924 07:06:45.400152 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:45.400156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:45.400254 (+ 98us) server_negotiation.cc:407] Connection header received
0924 07:06:45.400366 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:45.400369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:45.400410 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:45.400473 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:45.400690 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.401391 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.402890 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.403533 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.403630 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:45.403703 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:45.403957 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 07:06:45.404020 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:06:45.404148 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:45.399725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:51905 (local address 127.3.217.3:39493)
0924 07:06:45.399839 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:45.399844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:45.399858 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:06:45.400146 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:45.400150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:45.400199 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:45.400272 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:06:45.400521 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.401303 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.402890 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.403534 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.403630 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:45.403703 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:45.403957 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 07:06:45.404020 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:06:45.404166 14507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:45.399657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:39493 (local address 127.3.217.2:51905)
0924 07:06:45.399778 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:45.399794 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:45.399894 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:45.400295 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:45.400298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:45.400311 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:45.400432 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:45.400437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.401411 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.401413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:45.402762 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:45.402773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.403669 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.403673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:45.403805 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:45.403854 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 07:06:45.403918 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 07:06:45.404014 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":87}
I20260924 07:06:45.404177 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:45.399910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:35489 (local address 127.3.217.2:59515)
0924 07:06:45.400174 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:45.400186 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:06:45.400284 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:06:45.400498 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:06:45.400500 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:06:45.400526 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:06:45.400608 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:45.400612 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.401474 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.401477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:45.402762 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:06:45.402773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.403669 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.403673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:06:45.403800 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:45.403843 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 07:06:45.403924 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 07:06:45.404014 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":78,"thread_start_us":101,"threads_started":1}
I20260924 07:06:45.404855 14324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a403dfa59e415ea4ff702bb767fb94" candidate_uuid: "1b9c5c1547b34195976b56a3a591d875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3e687a9a16048658a8ecaf90f3c15e3" is_pre_election: true
I20260924 07:06:45.404829 14593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a403dfa59e415ea4ff702bb767fb94" candidate_uuid: "1b9c5c1547b34195976b56a3a591d875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7094b3bdb804789b8ba9d2b43f78f5d" is_pre_election: true
I20260924 07:06:45.405107 14324 raft_consensus.cc:2454] T 63a403dfa59e415ea4ff702bb767fb94 P e3e687a9a16048658a8ecaf90f3c15e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b9c5c1547b34195976b56a3a591d875 in term 0.
I20260924 07:06:45.405121 14593 raft_consensus.cc:2454] T 63a403dfa59e415ea4ff702bb767fb94 P f7094b3bdb804789b8ba9d2b43f78f5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b9c5c1547b34195976b56a3a591d875 in term 0.
I20260924 07:06:45.405524 14394 leader_election.cc:302] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b9c5c1547b34195976b56a3a591d875, e3e687a9a16048658a8ecaf90f3c15e3; no voters:
I20260924 07:06:45.405771 14664 raft_consensus.cc:2781] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:06:45.405838 14664 raft_consensus.cc:482] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:06:45.405869 14664 raft_consensus.cc:3037] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:45.406880 14664 raft_consensus.cc:504] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7094b3bdb804789b8ba9d2b43f78f5d" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 39493 } } peers { permanent_uuid: "1b9c5c1547b34195976b56a3a591d875" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 46201 } } peers { permanent_uuid: "e3e687a9a16048658a8ecaf90f3c15e3" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35489 } }
I20260924 07:06:45.407583 14664 leader_election.cc:288] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875 [CANDIDATE]: Term 1 election: Requested vote from peers f7094b3bdb804789b8ba9d2b43f78f5d (127.3.217.3:39493), e3e687a9a16048658a8ecaf90f3c15e3 (127.3.217.1:35489)
I20260924 07:06:45.407953 14593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a403dfa59e415ea4ff702bb767fb94" candidate_uuid: "1b9c5c1547b34195976b56a3a591d875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7094b3bdb804789b8ba9d2b43f78f5d"
I20260924 07:06:45.408079 14593 raft_consensus.cc:3037] T 63a403dfa59e415ea4ff702bb767fb94 P f7094b3bdb804789b8ba9d2b43f78f5d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:45.408206 14324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a403dfa59e415ea4ff702bb767fb94" candidate_uuid: "1b9c5c1547b34195976b56a3a591d875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3e687a9a16048658a8ecaf90f3c15e3"
I20260924 07:06:45.408473 14324 raft_consensus.cc:3037] T 63a403dfa59e415ea4ff702bb767fb94 P e3e687a9a16048658a8ecaf90f3c15e3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:06:45.409121 14593 raft_consensus.cc:2454] T 63a403dfa59e415ea4ff702bb767fb94 P f7094b3bdb804789b8ba9d2b43f78f5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b9c5c1547b34195976b56a3a591d875 in term 1.
I20260924 07:06:45.409366 14394 leader_election.cc:302] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b9c5c1547b34195976b56a3a591d875, f7094b3bdb804789b8ba9d2b43f78f5d; no voters:
I20260924 07:06:45.409516 14324 raft_consensus.cc:2454] T 63a403dfa59e415ea4ff702bb767fb94 P e3e687a9a16048658a8ecaf90f3c15e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b9c5c1547b34195976b56a3a591d875 in term 1.
I20260924 07:06:45.409700 14664 raft_consensus.cc:2781] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:06:45.409898 14664 raft_consensus.cc:686] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875 [term 1 LEADER]: Becoming Leader. State: Replica: 1b9c5c1547b34195976b56a3a591d875, State: Running, Role: LEADER
I20260924 07:06:45.410219 14664 consensus_queue.cc:237] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7094b3bdb804789b8ba9d2b43f78f5d" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 39493 } } peers { permanent_uuid: "1b9c5c1547b34195976b56a3a591d875" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 46201 } } peers { permanent_uuid: "e3e687a9a16048658a8ecaf90f3c15e3" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35489 } }
I20260924 07:06:45.413646 14176 catalog_manager.cc:5495] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b9c5c1547b34195976b56a3a591d875 (127.3.217.2). New cstate: current_term: 1 leader_uuid: "1b9c5c1547b34195976b56a3a591d875" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7094b3bdb804789b8ba9d2b43f78f5d" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 39493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b9c5c1547b34195976b56a3a591d875" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 46201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3e687a9a16048658a8ecaf90f3c15e3" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35489 } health_report { overall_health: UNKNOWN } } }
W20260924 07:06:45.416422 14506 tablet.cc:2369] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:06:45.432966 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:45.429685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46868 (local address 127.3.217.2:46201)
0924 07:06:45.429803 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 07:06:45.429808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:06:45.429821 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:06:45.429896 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:06:45.429898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:06:45.429960 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:06:45.430048 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:06:45.430263 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.430775 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.431632 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:06:45.431844 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:06:45.431930 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:06:45.432205 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:06:45.432599 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:06:45.432699 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 07:06:45.432769 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 07:06:45.432824 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:06:45.438499 14324 raft_consensus.cc:1261] T 63a403dfa59e415ea4ff702bb767fb94 P e3e687a9a16048658a8ecaf90f3c15e3 [term 1 FOLLOWER]: Refusing update from remote peer 1b9c5c1547b34195976b56a3a591d875: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:06:45.438515 14593 raft_consensus.cc:1261] T 63a403dfa59e415ea4ff702bb767fb94 P f7094b3bdb804789b8ba9d2b43f78f5d [term 1 FOLLOWER]: Refusing update from remote peer 1b9c5c1547b34195976b56a3a591d875: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:06:45.439136 14664 consensus_queue.cc:1035] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3e687a9a16048658a8ecaf90f3c15e3" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 35489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:06:45.439321 14666 consensus_queue.cc:1035] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7094b3bdb804789b8ba9d2b43f78f5d" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 39493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:06:45.449056 14670 mvcc.cc:205] Tried to move back new op lower bound from 7332796847868911616 to 7332796847764070400. Current Snapshot: MvccSnapshot[applied={T|T < 7332796847868911616}]
I20260924 07:06:45.449219 14669 mvcc.cc:205] Tried to move back new op lower bound from 7332796847868911616 to 7332796847764070400. Current Snapshot: MvccSnapshot[applied={T|T < 7332796847868911616}]
I20260924 07:06:45.457691 14178 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:06:45.457849 14178 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:06:45.460041 14178 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac7b91b3f8884fc2a1c66e69be040866: Sending DeleteTablet for 3 replicas of tablet 63a403dfa59e415ea4ff702bb767fb94
I20260924 07:06:45.460786 14303 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a403dfa59e415ea4ff702bb767fb94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:45 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:39560
I20260924 07:06:45.460814 14573 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a403dfa59e415ea4ff702bb767fb94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:45 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:45558
I20260924 07:06:45.461186 14678 tablet_replica.cc:330] T 63a403dfa59e415ea4ff702bb767fb94 P e3e687a9a16048658a8ecaf90f3c15e3: stopping tablet replica
I20260924 07:06:45.461441 14678 raft_consensus.cc:2229] T 63a403dfa59e415ea4ff702bb767fb94 P e3e687a9a16048658a8ecaf90f3c15e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:06:45.461606 14439 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a403dfa59e415ea4ff702bb767fb94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:06:45 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:46864
I20260924 07:06:45.461709 14678 raft_consensus.cc:2258] T 63a403dfa59e415ea4ff702bb767fb94 P e3e687a9a16048658a8ecaf90f3c15e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:45.461988 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 14240
I20260924 07:06:45.461999 14679 tablet_replica.cc:330] T 63a403dfa59e415ea4ff702bb767fb94 P f7094b3bdb804789b8ba9d2b43f78f5d: stopping tablet replica
I20260924 07:06:45.462239 14679 raft_consensus.cc:2229] T 63a403dfa59e415ea4ff702bb767fb94 P f7094b3bdb804789b8ba9d2b43f78f5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:06:45.462491 14679 raft_consensus.cc:2258] T 63a403dfa59e415ea4ff702bb767fb94 P f7094b3bdb804789b8ba9d2b43f78f5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:45.462790 14680 tablet_replica.cc:330] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875: stopping tablet replica
I20260924 07:06:45.463025 14680 raft_consensus.cc:2229] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:06:45.463353 14680 raft_consensus.cc:2258] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:06:45.464932 14679 ts_tablet_manager.cc:1905] T 63a403dfa59e415ea4ff702bb767fb94 P f7094b3bdb804789b8ba9d2b43f78f5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:06:45.467979 14679 ts_tablet_manager.cc:1918] T 63a403dfa59e415ea4ff702bb767fb94 P f7094b3bdb804789b8ba9d2b43f78f5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:06:45.468075 14679 log.cc:1196] T 63a403dfa59e415ea4ff702bb767fb94 P f7094b3bdb804789b8ba9d2b43f78f5d: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/63a403dfa59e415ea4ff702bb767fb94
I20260924 07:06:45.468379 14679 ts_tablet_manager.cc:1939] T 63a403dfa59e415ea4ff702bb767fb94 P f7094b3bdb804789b8ba9d2b43f78f5d: Deleting consensus metadata
I20260924 07:06:45.468743 14680 ts_tablet_manager.cc:1905] T 63a403dfa59e415ea4ff702bb767fb94 P 1b9c5c1547b34195976b56a3a591d875: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 07:06:45.468742 14165 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.217.1:35489 (error 108)
I20260924 07:06:45.469923 14165 catalog_manager.cc:4841] TS f7094b3bdb804789b8ba9d2b43f78f5d (127.3.217.3:39493): tablet 63a403dfa59e415ea4ff702bb767fb94 (table test-table [id=dcb5fe4b80564e9391bee21c1101919c]) successfully deleted
I20260924 07:06:45.470108 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:45.469698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:35489 (local address 127.0.0.1:39566)
0924 07:06:45.469964 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:45.470029 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.1:35489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
W20260924 07:06:45.470275 14165 catalog_manager.cc:4584] TS e3e687a9a16048658a8ecaf90f3c15e3 (127.3.217.1:35489): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63a403dfa59e415ea4ff702bb767fb94: Network error: Client connection negotiation failed: client connection to 127.3.217.1:35489: connect: Connection refused (error 111)
I20260924 07:06:45.470553 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 14375
I20260924 07:06:45.475323 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 14510
I20260924 07:06:45.476616 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:06:45.475414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:46201 (local address 127.0.0.1:46876)
0924 07:06:45.476490 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0924 07:06:45.476543 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.2:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1004}
I20260924 07:06:45.480597 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 14146
2026-09-24T07:06:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2308 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:06:45 dist-test-slave-2p7j krb5kdc[14684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:06:45 dist-test-slave-2p7j krb5kdc[14684](info): set up 2 sockets
Sep 24 07:06:45 dist-test-slave-2p7j krb5kdc[14684](info): commencing operation
krb5kdc: starting...
W20260924 07:06:46.473068 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:06:46 dist-test-slave-2p7j krb5kdc[14684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:06:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:06:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.217.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.217.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.217.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/rangeradmin_127.3.217.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.217.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/rangeradmin_127.3.217.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.217.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.217.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.217.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/rangerlookup_127.3.217.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.217.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/rangerlookup_127.3.217.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.217.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.217.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.217.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/HTTP_127.3.217.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.217.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/HTTP_127.3.217.20@KRBTEST.COM.keytab.
I20260924 07:06:46.639211 3940 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskSXrqy4/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/postgres -l logfile start
2026-09-24 07:06:49.359 UTC [14714] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 07:06:49.360 UTC [14714] LOG: listening on IPv4 address "127.3.217.20", port 39629
2026-09-24 07:06:49.363 UTC [14714] LOG: listening on Unix socket "/tmp/.s.PGSQL.39629"
2026-09-24 07:06:49.383 UTC [14717] LOG: database system was shut down at 2026-09-24 07:06:47 UTC
2026-09-24 07:06:49.386 UTC [14714] LOG: database system is ready to accept connections
I20260924 07:06:50.280479 3940 mini_postgres.cc:96] Postgres bound to 39629
2026-09-24 07:06:50.287 UTC [14725] FATAL: database "slave" does not exist
127.3.217.20:39629 - accepting connections
I20260924 07:06:50.288193 3940 mini_ranger.cc:161] Starting Ranger...
I20260924 07:06:50.302937 3940 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 07:06:50.870028 3940 mini_ranger.cc:87] Created ranger Postgres database
I20260924 07:06:50.870114 3940 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ranger-admin
2026-09-24 07:06:51,189 [I] DB FLAVOR :POSTGRES
2026-09-24 07:06:51,190 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 07:06:51,190 [I] Checking connection..
2026-09-24 07:06:51,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 07:06:51,461 [I] Checking connection passed.
2026-09-24 07:06:51,461 [I] --------- Verifying version history table ---------
2026-09-24 07:06:51,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:06:51,704 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 07:06:51,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:06:51,946 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 07:06:51,946 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 07:06:51,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 07:06:52.227 UTC [14789] WARNING: there is no transaction in progress
2026-09-24 07:06:52,234 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 07:06:52,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:06:52,481 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 07:06:52,482 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 07:06:52,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 07:06:52,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 07:06:52,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2p7j.c.gcp-upstream.internal','N') ;"
2026-09-24 07:06:53,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 07:06:53,497 [I] Table x_portal_user does not exist in database ranger
2026-09-24 07:06:53,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 07:06:53,776 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 07:06:53,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 07:06:54,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 07:06:54,283 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 07:06:54,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 07:06:55.294 UTC [14898] WARNING: there is no transaction in progress
2026-09-24 07:06:55.313 UTC [14898] WARNING: there is no transaction in progress
2026-09-24 07:06:55.332 UTC [14898] WARNING: there is no transaction in progress
2026-09-24 07:06:55.348 UTC [14898] WARNING: there is no transaction in progress
2026-09-24 07:06:55.368 UTC [14898] WARNING: there is no transaction in progress
2026-09-24 07:06:55.387 UTC [14898] WARNING: there is no transaction in progress
2026-09-24 07:06:55.444 UTC [14898] WARNING: there is no transaction in progress
2026-09-24 07:06:55.452 UTC [14898] WARNING: there is no transaction in progress
2026-09-24 07:06:55.467 UTC [14898] WARNING: there is no transaction in progress
2026-09-24 07:06:55.477 UTC [14898] WARNING: there is no transaction in progress
2026-09-24 07:06:55.487 UTC [14898] WARNING: there is no transaction in progress
2026-09-24 07:06:55.501 UTC [14898] WARNING: there is no transaction in progress
2026-09-24 07:06:55.513 UTC [14898] WARNING: there is no transaction in progress
2026-09-24 07:06:55.519 UTC [14898] WARNING: there is no transaction in progress
2026-09-24 07:06:55.527 UTC [14898] WARNING: there is no transaction in progress
2026-09-24 07:06:55.535 UTC [14898] WARNING: there is no transaction in progress
2026-09-24 07:06:56,210 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 07:06:56,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 07:06:56,461 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 07:06:56,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 07:06:56,694 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 07:06:56,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 07:06:56,954 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 07:06:56,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 07:06:57,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 07:06:57,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2p7j.c.gcp-upstream.internal';"
2026-09-24 07:06:57,661 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 07:06:57,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 07:06:57,898 [I] DB_PATCHES have already been applied
I20260924 07:06:57.904103 3940 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ranger-admin:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskSXrqy4/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 07:06:57.904163 3940 mini_ranger.cc:193] Using host: 127.3.217.20
I20260924 07:06:57.907323 3940 mini_ranger.cc:227] Ranger admin URL: http://127.3.217.20:35401
Sep 24, 2026 7:06:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:06:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:06:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 7:06:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 7:06:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 7:06:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 7:06:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 07:06:58.919677 3940 mini_ranger.cc:160] Time spent starting Ranger: real 8.631s user 0.000s sys 0.009s
Sep 24, 2026 7:07:19 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 7:07:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 7:07:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 7:07:20 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 7:07:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 7:07:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 7:07:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 7:07:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 7:07:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 7:07:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 7:07:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 7:07:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 7:07:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 7:07:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 7:07:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 7:07:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 7:07:20 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 7:07:20 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 07:07:21.490887 3940 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.217.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.217.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.217.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/rangerkms_127.3.217.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.217.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/rangerkms_127.3.217.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.217.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.217.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.217.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/HTTP_127.3.217.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.217.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/HTTP_127.3.217.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 07:07:21 dist-test-slave-2p7j krb5kdc[14684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233641, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 07:07:21.594292 3940 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 07:07:21.604074 3940 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 07:07:22.630940 3940 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 07:07:22.631032 3940 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ranger-kms
I20260924 07:07:22.631129 3940 mini_ranger_kms.cc:227] Using postgres at 127.3.217.20:39629
2026-09-24 07:07:22,835 [I] DB FLAVOR :POSTGRES
2026-09-24 07:07:22,835 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 07:07:22,835 [I] Checking connection
2026-09-24 07:07:22,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:39629/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 07:07:23,119 [I] connection success
2026-09-24 07:07:23,119 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 07:07:23,119 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 07:07:23,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:39629/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 07:07:23,406 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 07:07:23,406 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 07:07:23,407 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 07:07:23,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:39629/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 07:07:23,682 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 07:07:24.179610 3940 mini_ranger_kms.cc:310] Created kms service
I20260924 07:07:24.380625 3940 mini_ranger_kms.cc:326] Created kudu user
I20260924 07:07:24.420421 3940 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 07:07:24 dist-test-slave-2p7j krb5kdc[14684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233641, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.217.20@KRBTEST.COM
I20260924 07:07:24.811072 3940 mini_ranger_kms.cc:384] Added ranger policy
I20260924 07:07:24.811234 3940 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ranger-kms:/tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 07:07:24.811259 3940 mini_ranger_kms.cc:240] Using host: 127.3.217.20
I20260924 07:07:24.816336 3940 mini_ranger_kms.cc:276] Ranger KMS PID: 15079
I20260924 07:07:24.816455 3940 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.217.20:46645
Sep 24, 2026 7:07:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:07:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:07:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 7:07:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 7:07:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 7:07:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 07:07:26.851603 3940 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.257s user 0.014s sys 0.169s
I20260924 07:07:26.851747 3940 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 07:07:26.851850 3940 mini_ranger_kms.cc:401] 127.3.217.20:46645/kms/v1/keys
Sep 24 07:07:27 dist-test-slave-2p7j krb5kdc[14684](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790233647, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.217.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 07:07:27 dist-test-slave-2p7j krb5kdc[14684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233647, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.217.20@KRBTEST.COM for HTTP/127.3.217.20@KRBTEST.COM
Sep 24 07:07:27 dist-test-slave-2p7j krb5kdc[14684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233647, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.217.20@KRBTEST.COM for HTTP/127.3.217.20@KRBTEST.COM
Sep 24 07:07:30 dist-test-slave-2p7j krb5kdc[14684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:07:30.308879 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:33039
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:33039
--ranger_config_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.217.20:46645/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:07:30.420312 15129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:07:30.420542 15129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:07:30.420578 15129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:07:30.424446 15129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:07:30.424491 15129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:07:30.424515 15129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:07:30.424535 15129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:07:30.424599 15129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:07:30.430886 15129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.217.20:46645/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:33039
--ranger_config_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:33039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:07:30.431775 15129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:07:30.432636 15129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:07:30.440372 15134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:07:30.440376 15135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:07:30.440447 15137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:07:30.440605 15129 server_base.cc:1034] running on GCE node
I20260924 07:07:30.441144 15129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:07:30.442101 15129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:07:30.443324 15129 hybrid_clock.cc:648] HybridClock initialized: now 1790233650443307 us; error 72 us; skew 500 ppm
Sep 24 07:07:30 dist-test-slave-2p7j krb5kdc[14684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:07:30.452111 15129 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:07:30.452943 15129 webserver.cc:460] Webserver started at http://127.3.217.62:36995/ using document root <none> and password file <none>
I20260924 07:07:30.453410 15129 fs_manager.cc:359] Metadata directory not provided
I20260924 07:07:30.453459 15129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:07:30.453610 15129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:07:30 dist-test-slave-2p7j krb5kdc[14684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for HTTP/127.3.217.20@KRBTEST.COM
I20260924 07:07:30.513957 15129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "5e6325ad640a4f61b25d6d5ed8b76ff7"
format_stamp: "Formatted at 2026-09-24 07:07:30 on dist-test-slave-2p7j"
server_key: "ccb89f63a402d0effb6bfd8950d3ff2c"
server_key_iv: "fb62a4204d67542eff3ed5670ce5ad82"
server_key_version: "kuduclusterkey@0"
I20260924 07:07:30.514624 15129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "5e6325ad640a4f61b25d6d5ed8b76ff7"
format_stamp: "Formatted at 2026-09-24 07:07:30 on dist-test-slave-2p7j"
server_key: "ccb89f63a402d0effb6bfd8950d3ff2c"
server_key_iv: "fb62a4204d67542eff3ed5670ce5ad82"
server_key_version: "kuduclusterkey@0"
I20260924 07:07:30.518174 15129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 07:07:30.535076 15144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:07:30.536324 15129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 07:07:30.536434 15129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "5e6325ad640a4f61b25d6d5ed8b76ff7"
format_stamp: "Formatted at 2026-09-24 07:07:30 on dist-test-slave-2p7j"
server_key: "ccb89f63a402d0effb6bfd8950d3ff2c"
server_key_iv: "fb62a4204d67542eff3ed5670ce5ad82"
server_key_version: "kuduclusterkey@0"
I20260924 07:07:30.536551 15129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:07:30.551321 15129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:07:30.553918 15129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:07:30.554076 15129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:07:30.561976 15129 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:33039
I20260924 07:07:30.562023 15196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:33039 every 8 connection(s)
I20260924 07:07:30.562860 15129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:07:30.565467 15197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:07:30.565884 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 15129
I20260924 07:07:30.566004 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:07:30.566282 3940 external_mini_cluster.cc:1428] Setting key e692b5498e28fac5d141d7a37af9d506
I20260924 07:07:30.571614 15197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e6325ad640a4f61b25d6d5ed8b76ff7: Bootstrap starting.
I20260924 07:07:30.573837 15197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e6325ad640a4f61b25d6d5ed8b76ff7: Neither blocks nor log segments found. Creating new log.
I20260924 07:07:30.574661 15197 log.cc:824] T 00000000000000000000000000000000 P 5e6325ad640a4f61b25d6d5ed8b76ff7: Log is configured to *not* fsync() on all Append() calls
Sep 24 07:07:30 dist-test-slave-2p7j krb5kdc[14684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:07:30.576607 15197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e6325ad640a4f61b25d6d5ed8b76ff7: No bootstrap required, opened a new log
I20260924 07:07:30.579553 15197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e6325ad640a4f61b25d6d5ed8b76ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e6325ad640a4f61b25d6d5ed8b76ff7" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33039 } }
I20260924 07:07:30.579771 15197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e6325ad640a4f61b25d6d5ed8b76ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:07:30.579838 15197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e6325ad640a4f61b25d6d5ed8b76ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e6325ad640a4f61b25d6d5ed8b76ff7, State: Initialized, Role: FOLLOWER
I20260924 07:07:30.580240 15197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e6325ad640a4f61b25d6d5ed8b76ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e6325ad640a4f61b25d6d5ed8b76ff7" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33039 } }
I20260924 07:07:30.580399 15197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e6325ad640a4f61b25d6d5ed8b76ff7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:07:30.580489 15197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e6325ad640a4f61b25d6d5ed8b76ff7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:07:30.580586 15197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e6325ad640a4f61b25d6d5ed8b76ff7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:07:30.581672 15197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e6325ad640a4f61b25d6d5ed8b76ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e6325ad640a4f61b25d6d5ed8b76ff7" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33039 } }
I20260924 07:07:30.582012 15197 leader_election.cc:302] T 00000000000000000000000000000000 P 5e6325ad640a4f61b25d6d5ed8b76ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e6325ad640a4f61b25d6d5ed8b76ff7; no voters:
I20260924 07:07:30.582298 15197 leader_election.cc:288] T 00000000000000000000000000000000 P 5e6325ad640a4f61b25d6d5ed8b76ff7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:07:30.582403 15202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e6325ad640a4f61b25d6d5ed8b76ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:07:30.582587 15202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e6325ad640a4f61b25d6d5ed8b76ff7 [term 1 LEADER]: Becoming Leader. State: Replica: 5e6325ad640a4f61b25d6d5ed8b76ff7, State: Running, Role: LEADER
I20260924 07:07:30.582926 15202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e6325ad640a4f61b25d6d5ed8b76ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e6325ad640a4f61b25d6d5ed8b76ff7" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33039 } }
I20260924 07:07:30.583228 15200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:07:30.567656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49260 (local address 127.3.217.62:33039)
0924 07:07:30.568072 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0924 07:07:30.568089 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:07:30.568129 (+ 40us) server_negotiation.cc:407] Connection header received
0924 07:07:30.568900 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:07:30.568921 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:07:30.569303 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:07:30.569623 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:07:30.570406 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:07:30.571828 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:07:30.573037 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:07:30.573761 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:07:30.573916 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:07:30.576919 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:07:30.576931 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:07:30.576958 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:07:30.577090 (+ 132us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:07:30.580158 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:07:30.580663 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:07:30.580669 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:07:30.580675 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:07:30.580800 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:07:30.581105 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:07:30.581108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:07:30.581110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:07:30.581597 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:07:30.581778 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 07:07:30.582089 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 07:07:30.582347 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":109,"threads_started":1}
I20260924 07:07:30.583371 15197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e6325ad640a4f61b25d6d5ed8b76ff7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:07:30.585242 15203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e6325ad640a4f61b25d6d5ed8b76ff7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e6325ad640a4f61b25d6d5ed8b76ff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e6325ad640a4f61b25d6d5ed8b76ff7" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33039 } } }
I20260924 07:07:30.585350 15203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e6325ad640a4f61b25d6d5ed8b76ff7 [sys.catalog]: This master's current role is: LEADER
I20260924 07:07:30.585459 15197 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 07:07:30.585632 15216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:07:30.585628 15204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e6325ad640a4f61b25d6d5ed8b76ff7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e6325ad640a4f61b25d6d5ed8b76ff7. Latest consensus state: current_term: 1 leader_uuid: "5e6325ad640a4f61b25d6d5ed8b76ff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e6325ad640a4f61b25d6d5ed8b76ff7" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33039 } } }
I20260924 07:07:30.585709 15204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e6325ad640a4f61b25d6d5ed8b76ff7 [sys.catalog]: This master's current role is: LEADER
I20260924 07:07:30.591861 15216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:07:30.598378 15216 catalog_manager.cc:1347] Generated new cluster ID: 92fe143e187243cd9ecc8e2342546f88
I20260924 07:07:30.598435 15216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:07:30.615478 15216 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:07:30.616369 15216 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:07:30.629909 15216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e6325ad640a4f61b25d6d5ed8b76ff7: Generated new TSK 0
I20260924 07:07:30.630600 15216 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
I20260924 07:07:31.295064 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.217.20:46645/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:33039
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:07:31.440088 15242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:07:31.440375 15242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:07:31.440438 15242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:07:31.444701 15242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:07:31.444754 15242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:07:31.444832 15242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:07:31.450026 15242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.217.20:46645/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:33039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:07:31.450950 15242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:07:31.451917 15242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:07:31.461154 15250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:07:31.461535 15247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:07:31.461650 15242 server_base.cc:1034] running on GCE node
W20260924 07:07:31.461789 15248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:07:31.462173 15242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:07:31.462731 15242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:07:31.463904 15242 hybrid_clock.cc:648] HybridClock initialized: now 1790233651463884 us; error 35 us; skew 500 ppm
Sep 24 07:07:31 dist-test-slave-2p7j krb5kdc[14684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:07:31.467156 15242 init.cc:377] Logged in from keytab as kudu/127.3.217.1@KRBTEST.COM (short username kudu)
I20260924 07:07:31.468178 15242 webserver.cc:460] Webserver started at http://127.3.217.1:37847/ using document root <none> and password file <none>
I20260924 07:07:31.468902 15242 fs_manager.cc:359] Metadata directory not provided
I20260924 07:07:31.469049 15242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:07:31.469301 15242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:07:31 dist-test-slave-2p7j krb5kdc[14684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for HTTP/127.3.217.20@KRBTEST.COM
I20260924 07:07:31.491276 15242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "ba5ce66107fe448d921237d58fefacd6"
format_stamp: "Formatted at 2026-09-24 07:07:31 on dist-test-slave-2p7j"
server_key: "a1e3fb351fcc10f8d4bbd4ad17340266"
server_key_iv: "99973f100ac122592ad0510f528a34c1"
server_key_version: "kuduclusterkey@0"
I20260924 07:07:31.493475 15242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "ba5ce66107fe448d921237d58fefacd6"
format_stamp: "Formatted at 2026-09-24 07:07:31 on dist-test-slave-2p7j"
server_key: "a1e3fb351fcc10f8d4bbd4ad17340266"
server_key_iv: "99973f100ac122592ad0510f528a34c1"
server_key_version: "kuduclusterkey@0"
I20260924 07:07:31.498706 15242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260924 07:07:31.519989 15259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:07:31.521623 15242 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:07:31.521812 15242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "ba5ce66107fe448d921237d58fefacd6"
format_stamp: "Formatted at 2026-09-24 07:07:31 on dist-test-slave-2p7j"
server_key: "a1e3fb351fcc10f8d4bbd4ad17340266"
server_key_iv: "99973f100ac122592ad0510f528a34c1"
server_key_version: "kuduclusterkey@0"
I20260924 07:07:31.522029 15242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:07:31.546407 15242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:07:31.550107 15242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:07:31.550346 15242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:07:31.551079 15242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:07:31.552135 15242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:07:31.552237 15242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:07:31.552359 15242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:07:31.552438 15242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:07:31.573227 15242 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:33667
I20260924 07:07:31.576627 15242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:07:31.580415 15372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:33667 every 8 connection(s)
I20260924 07:07:31.581868 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 15242
I20260924 07:07:31.581988 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:07:31.582288 3940 external_mini_cluster.cc:1428] Setting key 8bc9d11f35e63ad2fe91fe873d1e284c
Sep 24 07:07:31 dist-test-slave-2p7j krb5kdc[14684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.217.2@KRBTEST.COM; defaulting to no policy
I20260924 07:07:31.599995 15376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:07:31.584860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:47127 (local address 127.3.217.62:33039)
0924 07:07:31.589663 (+ 4803us) server_negotiation.cc:206] Beginning negotiation
0924 07:07:31.589671 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:07:31.589684 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:07:31.589841 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:07:31.589847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:07:31.589986 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:07:31.590093 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:07:31.591471 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:07:31.592081 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:07:31.593261 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:07:31.593487 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:07:31.593572 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:07:31.595794 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:07:31.595798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:07:31.595801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:07:31.595823 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:07:31.597157 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:07:31.597632 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:07:31.597635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:07:31.597637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:07:31.597719 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:07:31.598018 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:07:31.598021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:07:31.598023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:07:31.598224 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:07:31.598327 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 07:07:31.599634 (+ 1307us) server_negotiation.cc:299] Negotiation successful
0924 07:07:31.599759 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4731,"thread_start_us":97,"threads_started":1}
I20260924 07:07:31.600054 15375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:07:31.583342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:33039 (local address 127.3.217.1:47127)
0924 07:07:31.588221 (+ 4879us) negotiation.cc:107] Waiting for socket to connect
0924 07:07:31.588275 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 07:07:31.589272 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:07:31.590303 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:07:31.590312 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:07:31.590652 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:07:31.591329 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:07:31.591351 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:07:31.592203 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:07:31.592206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:07:31.593144 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:07:31.593154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:07:31.593620 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:07:31.593624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:07:31.593799 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:07:31.594330 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:07:31.594358 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:07:31.595643 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:07:31.597273 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:07:31.597278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:07:31.597293 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:07:31.597522 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:07:31.597805 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:07:31.597809 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:07:31.597810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:07:31.597931 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:07:31.598383 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:07:31.598389 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:07:31.598701 (+ 312us) client_negotiation.cc:769] Sending connection context
0924 07:07:31.598902 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 07:07:31.599184 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4661,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.2@KRBTEST.COM" created.
I20260924 07:07:31.601743 15373 heartbeater.cc:347] Connected to a master server at 127.3.217.62:33039
I20260924 07:07:31.601995 15373 heartbeater.cc:464] Registering TS with master...
I20260924 07:07:31.602497 15373 heartbeater.cc:511] Master 127.3.217.62:33039 requested a full tablet report, sending...
I20260924 07:07:31.604100 15161 ts_manager.cc:194] Registered new tserver with Master: ba5ce66107fe448d921237d58fefacd6 (127.3.217.1:33667)
I20260924 07:07:31.605512 15161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.1@KRBTEST.COM'} at 127.3.217.1:47127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
I20260924 07:07:31.618775 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.217.20:46645/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:33039
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:07:31 dist-test-slave-2p7j krb5kdc[14684](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790233651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 07:07:31.744344 15381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:07:31.744784 15381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:07:31.744880 15381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:07:31.748981 15381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:07:31.749042 15381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:07:31.749118 15381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:07:31.755627 15381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.217.20:46645/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:33039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:07:31.756567 15381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:07:31.757629 15381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:07:31.767515 15390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:07:31.767851 15387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:07:31.767948 15388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:07:31.768395 15381 server_base.cc:1034] running on GCE node
I20260924 07:07:31.768774 15381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:07:31.769356 15381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:07:31.772819 15381 hybrid_clock.cc:648] HybridClock initialized: now 1790233651772783 us; error 41 us; skew 500 ppm
Sep 24 07:07:31 dist-test-slave-2p7j krb5kdc[14684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:07:31.776788 15381 init.cc:377] Logged in from keytab as kudu/127.3.217.2@KRBTEST.COM (short username kudu)
I20260924 07:07:31.777609 15381 webserver.cc:460] Webserver started at http://127.3.217.2:45981/ using document root <none> and password file <none>
I20260924 07:07:31.778116 15381 fs_manager.cc:359] Metadata directory not provided
I20260924 07:07:31.778178 15381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:07:31.778366 15381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:07:31 dist-test-slave-2p7j krb5kdc[14684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for HTTP/127.3.217.20@KRBTEST.COM
I20260924 07:07:31.801414 15381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "ba2b320a6d3d4a9dace89b01a16ed1ac"
format_stamp: "Formatted at 2026-09-24 07:07:31 on dist-test-slave-2p7j"
server_key: "c08459f31b991704880ef2abd7a6bd96"
server_key_iv: "7eb00a0ef71cf6c04b029fdfb5d31ee3"
server_key_version: "kuduclusterkey@0"
I20260924 07:07:31.801983 15381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "ba2b320a6d3d4a9dace89b01a16ed1ac"
format_stamp: "Formatted at 2026-09-24 07:07:31 on dist-test-slave-2p7j"
server_key: "c08459f31b991704880ef2abd7a6bd96"
server_key_iv: "7eb00a0ef71cf6c04b029fdfb5d31ee3"
server_key_version: "kuduclusterkey@0"
I20260924 07:07:31.805682 15381 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:07:31.828065 15397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:07:31.829662 15381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:07:31.829782 15381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "ba2b320a6d3d4a9dace89b01a16ed1ac"
format_stamp: "Formatted at 2026-09-24 07:07:31 on dist-test-slave-2p7j"
server_key: "c08459f31b991704880ef2abd7a6bd96"
server_key_iv: "7eb00a0ef71cf6c04b029fdfb5d31ee3"
server_key_version: "kuduclusterkey@0"
I20260924 07:07:31.829907 15381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:07:31.861343 15381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:07:31.864725 15381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:07:31.864943 15381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:07:31.865633 15381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:07:31.866668 15381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:07:31.866765 15381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:07:31.866856 15381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:07:31.866915 15381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:07:31.877584 15381 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:37977
I20260924 07:07:31.878697 15381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:07:31.882586 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 15381
I20260924 07:07:31.882704 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:07:31.883004 3940 external_mini_cluster.cc:1428] Setting key eaae73d931b33d2ea224d881fd8c97bc
I20260924 07:07:31.896440 15510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:37977 every 8 connection(s)
Sep 24 07:07:31 dist-test-slave-2p7j krb5kdc[14684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:07:31.917795 15376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:07:31.895372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:51099 (local address 127.3.217.62:33039)
0924 07:07:31.895532 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 07:07:31.895537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:07:31.897038 (+ 1501us) server_negotiation.cc:407] Connection header received
0924 07:07:31.900736 (+ 3698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:07:31.900740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:07:31.900813 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:07:31.900876 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:07:31.902536 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:07:31.903360 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:07:31.908363 (+ 5003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:07:31.908735 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:07:31.908822 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:07:31.913482 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:07:31.913485 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:07:31.913488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:07:31.913516 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:07:31.915350 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:07:31.915990 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:07:31.915994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:07:31.915996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:07:31.916085 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:07:31.916457 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:07:31.916460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:07:31.916462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:07:31.916668 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:07:31.916747 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 07:07:31.917479 (+ 732us) server_negotiation.cc:299] Negotiation successful
0924 07:07:31.917616 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
WARNING: no policy specified for kudu/127.3.217.3@KRBTEST.COM; defaulting to no policy
I20260924 07:07:31.920470 15514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:07:31.894289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:33039 (local address 127.3.217.2:51099)
0924 07:07:31.896880 (+ 2591us) negotiation.cc:107] Waiting for socket to connect
0924 07:07:31.896934 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 07:07:31.898016 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:07:31.901295 (+ 3279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:07:31.901305 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:07:31.901657 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:07:31.902375 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:07:31.902393 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:07:31.903547 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:07:31.903551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:07:31.904965 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:07:31.904976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:07:31.909021 (+ 4045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:07:31.909025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:07:31.909237 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:07:31.911639 (+ 2402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:07:31.911660 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:07:31.913296 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:07:31.915490 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:07:31.915496 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:07:31.915531 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:07:31.915864 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:07:31.916187 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:07:31.916189 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:07:31.916191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:07:31.916353 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:07:31.916813 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:07:31.916819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:07:31.917177 (+ 358us) client_negotiation.cc:769] Sending connection context
0924 07:07:31.918261 (+ 1084us) client_negotiation.cc:240] Negotiation successful
0924 07:07:31.918600 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2341,"thread_start_us":149,"threads_started":1}
I20260924 07:07:31.922032 15511 heartbeater.cc:347] Connected to a master server at 127.3.217.62:33039
I20260924 07:07:31.922259 15511 heartbeater.cc:464] Registering TS with master...
I20260924 07:07:31.922838 15511 heartbeater.cc:511] Master 127.3.217.62:33039 requested a full tablet report, sending...
I20260924 07:07:31.924001 15161 ts_manager.cc:194] Registered new tserver with Master: ba2b320a6d3d4a9dace89b01a16ed1ac (127.3.217.2:37977)
I20260924 07:07:31.924695 15161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.2@KRBTEST.COM'} at 127.3.217.2:51099
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
I20260924 07:07:31.962723 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.217.20:46645/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:33039
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:07:32.124311 15516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:07:32.124718 15516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:07:32.124845 15516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:07:32.130878 15516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:07:32.131016 15516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:07:32.131138 15516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:07:32.139429 15516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.217.20:46645/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:33039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:07:32.141069 15516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:07:32.142812 15516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:07:32.155974 15522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:07:32.156415 15516 server_base.cc:1034] running on GCE node
W20260924 07:07:32.156692 15524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:07:32.156941 15521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:07:32.157414 15516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:07:32.158098 15516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:07:32.159265 15516 hybrid_clock.cc:648] HybridClock initialized: now 1790233652159209 us; error 70 us; skew 500 ppm
Sep 24 07:07:32 dist-test-slave-2p7j krb5kdc[14684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233652, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:07:32.163515 15516 init.cc:377] Logged in from keytab as kudu/127.3.217.3@KRBTEST.COM (short username kudu)
I20260924 07:07:32.164588 15516 webserver.cc:460] Webserver started at http://127.3.217.3:40933/ using document root <none> and password file <none>
I20260924 07:07:32.165272 15516 fs_manager.cc:359] Metadata directory not provided
I20260924 07:07:32.165414 15516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:07:32.165688 15516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:07:32 dist-test-slave-2p7j krb5kdc[14684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233652, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for HTTP/127.3.217.20@KRBTEST.COM
I20260924 07:07:32.183486 15516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "013c9e2171f34b3fb438e6a80f7c9f53"
format_stamp: "Formatted at 2026-09-24 07:07:32 on dist-test-slave-2p7j"
server_key: "047c0e65f23a94a120f5b7f5316e5dcc"
server_key_iv: "0310656d7aadee6ad212c6dc1a8e99b2"
server_key_version: "kuduclusterkey@0"
I20260924 07:07:32.185125 15516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "013c9e2171f34b3fb438e6a80f7c9f53"
format_stamp: "Formatted at 2026-09-24 07:07:32 on dist-test-slave-2p7j"
server_key: "047c0e65f23a94a120f5b7f5316e5dcc"
server_key_iv: "0310656d7aadee6ad212c6dc1a8e99b2"
server_key_version: "kuduclusterkey@0"
I20260924 07:07:32.189693 15516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:07:32.210057 15531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:07:32.211686 15516 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 07:07:32.211805 15516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "013c9e2171f34b3fb438e6a80f7c9f53"
format_stamp: "Formatted at 2026-09-24 07:07:32 on dist-test-slave-2p7j"
server_key: "047c0e65f23a94a120f5b7f5316e5dcc"
server_key_iv: "0310656d7aadee6ad212c6dc1a8e99b2"
server_key_version: "kuduclusterkey@0"
I20260924 07:07:32.211921 15516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:07:32.235668 15516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:07:32.239406 15516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:07:32.239676 15516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:07:32.240424 15516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:07:32.241460 15516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:07:32.241566 15516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:07:32.241665 15516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:07:32.241724 15516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:07:32.260875 15516 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:40571
I20260924 07:07:32.262429 15516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:07:32.263612 15644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:40571 every 8 connection(s)
I20260924 07:07:32.265705 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 15516
I20260924 07:07:32.265830 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:07:32.266096 3940 external_mini_cluster.cc:1428] Setting key 2e56244fd810be8b0adf9ddf1b4477e6
Sep 24 07:07:32 dist-test-slave-2p7j krb5kdc[14684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233652, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:07:32.281414 15376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:07:32.267489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:43063 (local address 127.3.217.62:33039)
0924 07:07:32.267620 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:07:32.267625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:07:32.268206 (+ 581us) server_negotiation.cc:407] Connection header received
0924 07:07:32.269381 (+ 1175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:07:32.269385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:07:32.269434 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:07:32.269482 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:07:32.271165 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.271951 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.273609 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.273954 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.274027 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:07:32.277122 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:07:32.277126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:07:32.277129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:07:32.277167 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:07:32.279052 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:07:32.279683 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:07:32.279687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:07:32.279689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:07:32.279786 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:07:32.280183 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:07:32.280186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:07:32.280188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:07:32.280394 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:07:32.280470 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:07:32.281125 (+ 655us) server_negotiation.cc:299] Negotiation successful
0924 07:07:32.281248 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 07:07:32.283119 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:07:32.266444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:33039 (local address 127.3.217.3:43063)
0924 07:07:32.268068 (+ 1624us) negotiation.cc:107] Waiting for socket to connect
0924 07:07:32.268120 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 07:07:32.269148 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:07:32.269859 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:07:32.269867 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:07:32.270198 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:07:32.271014 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:07:32.271035 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.272097 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.272100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:07:32.273477 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:07:32.273489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.274236 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.274239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:07:32.274437 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:07:32.275100 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:07:32.275120 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:07:32.276938 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:07:32.279191 (+ 2253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:07:32.279197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:07:32.279211 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:07:32.279570 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:07:32.279894 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:07:32.279897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:07:32.279899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:07:32.280076 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:07:32.280533 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:07:32.280539 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:07:32.280861 (+ 322us) client_negotiation.cc:769] Sending connection context
0924 07:07:32.281829 (+ 968us) client_negotiation.cc:240] Negotiation successful
0924 07:07:32.282137 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1417,"thread_start_us":138,"threads_started":1}
I20260924 07:07:32.284509 15645 heartbeater.cc:347] Connected to a master server at 127.3.217.62:33039
I20260924 07:07:32.284722 15645 heartbeater.cc:464] Registering TS with master...
I20260924 07:07:32.285228 15645 heartbeater.cc:511] Master 127.3.217.62:33039 requested a full tablet report, sending...
I20260924 07:07:32.286269 15161 ts_manager.cc:194] Registered new tserver with Master: 013c9e2171f34b3fb438e6a80f7c9f53 (127.3.217.3:40571)
I20260924 07:07:32.286985 15161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.3@KRBTEST.COM'} at 127.3.217.3:43063
I20260924 07:07:32.291254 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 07:07:32 dist-test-slave-2p7j krb5kdc[14684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for HTTP/127.3.217.20@KRBTEST.COM
I20260924 07:07:32.305364 15376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:07:32.295651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49274 (local address 127.3.217.62:33039)
0924 07:07:32.295760 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 07:07:32.295764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:07:32.296363 (+ 599us) server_negotiation.cc:407] Connection header received
0924 07:07:32.296535 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:07:32.296538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:07:32.296582 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:07:32.296630 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:07:32.297538 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.298365 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.299578 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.299925 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.299996 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:07:32.301645 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:07:32.301648 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:07:32.301651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:07:32.301675 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:07:32.303270 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:07:32.304058 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:07:32.304061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:07:32.304063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:07:32.304146 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:07:32.304437 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:07:32.304440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:07:32.304442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:07:32.304644 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:07:32.304754 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 07:07:32.305106 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 07:07:32.305215 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:07:32.308722 15161 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:07:32.311916 15161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:07:32.328871 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:07:32.322285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:33667 (local address 127.0.0.1:54052)
0924 07:07:32.322646 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0924 07:07:32.322695 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 07:07:32.322804 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:07:32.324304 (+ 1500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:07:32.324310 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:07:32.324332 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:07:32.324451 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:07:32.324457 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.325748 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.325751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:07:32.327043 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:07:32.327051 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.328305 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.328309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:07:32.328404 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:07:32.328443 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 07:07:32.328531 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 07:07:32.328729 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":92,"threads_started":1}
I20260924 07:07:32.329818 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:07:32.322627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:40571 (local address 127.0.0.1:50128)
0924 07:07:32.323909 (+ 1282us) negotiation.cc:107] Waiting for socket to connect
0924 07:07:32.323921 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:07:32.323996 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:07:32.325473 (+ 1477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:07:32.325475 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:07:32.325490 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:07:32.325563 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:07:32.325568 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.326890 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.326893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:07:32.328307 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:07:32.328314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.329468 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.329471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:07:32.329557 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:07:32.329577 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 07:07:32.329631 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 07:07:32.329685 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1202,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260924 07:07:32.329813 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:07:32.323284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54052 (local address 127.3.217.1:33667)
0924 07:07:32.323679 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0924 07:07:32.323686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:07:32.323708 (+ 22us) server_negotiation.cc:407] Connection header received
0924 07:07:32.323781 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:07:32.323787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:07:32.324006 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:07:32.324138 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:07:32.324580 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.325634 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.327166 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.328023 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.328114 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:07:32.328241 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 07:07:32.329579 (+ 1338us) server_negotiation.cc:299] Negotiation successful
0924 07:07:32.329679 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":84,"threads_started":1}
I20260924 07:07:32.330418 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:07:32.323378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50128 (local address 127.3.217.3:40571)
0924 07:07:32.324463 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0924 07:07:32.324474 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:07:32.324494 (+ 20us) server_negotiation.cc:407] Connection header received
0924 07:07:32.325061 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:07:32.325068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:07:32.325234 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:07:32.325362 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:07:32.325671 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.326761 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.328444 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.329205 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.329299 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:07:32.329417 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 07:07:32.330202 (+ 785us) server_negotiation.cc:299] Negotiation successful
0924 07:07:32.330288 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":995,"thread_start_us":147,"threads_started":1}
I20260924 07:07:32.332801 15575 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa912dd467e49869dac1ec178da50cb (DEFAULT_TABLE table=test-table [id=e1fddfd2dcde497a85cbb7d13a856e8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:07:32.332796 15307 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa912dd467e49869dac1ec178da50cb (DEFAULT_TABLE table=test-table [id=e1fddfd2dcde497a85cbb7d13a856e8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:07:32.333849 15575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa912dd467e49869dac1ec178da50cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:07:32.333849 15307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa912dd467e49869dac1ec178da50cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:07:32.344897 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:07:32.322944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:37977 (local address 127.0.0.1:39856)
0924 07:07:32.329403 (+ 6459us) negotiation.cc:107] Waiting for socket to connect
0924 07:07:32.329415 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:07:32.329474 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:07:32.337641 (+ 8167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:07:32.337644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:07:32.337660 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:07:32.337733 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:07:32.337739 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.339047 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.339051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:07:32.340132 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:07:32.340141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.343687 (+ 3546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.343691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:07:32.344188 (+ 497us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:07:32.344218 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 07:07:32.344293 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 07:07:32.344404 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6409,"thread_start_us":73,"threads_started":1}
I20260924 07:07:32.345855 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:07:32.323129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39856 (local address 127.3.217.2:37977)
0924 07:07:32.337066 (+ 13937us) server_negotiation.cc:206] Beginning negotiation
0924 07:07:32.337080 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:07:32.337104 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:07:32.337177 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:07:32.337182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:07:32.337362 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:07:32.337504 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:07:32.337874 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.338921 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.340835 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.343357 (+ 2522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.343455 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:07:32.343624 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 07:07:32.345332 (+ 1708us) server_negotiation.cc:299] Negotiation successful
0924 07:07:32.345434 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13659,"thread_start_us":125,"threads_started":1}
I20260924 07:07:32.348892 15445 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa912dd467e49869dac1ec178da50cb (DEFAULT_TABLE table=test-table [id=e1fddfd2dcde497a85cbb7d13a856e8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:07:32.349728 15445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa912dd467e49869dac1ec178da50cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:07:32.352917 15664 tablet_bootstrap.cc:492] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53: Bootstrap starting.
I20260924 07:07:32.354718 15663 tablet_bootstrap.cc:492] T 9aa912dd467e49869dac1ec178da50cb P ba5ce66107fe448d921237d58fefacd6: Bootstrap starting.
I20260924 07:07:32.357077 15663 tablet_bootstrap.cc:654] T 9aa912dd467e49869dac1ec178da50cb P ba5ce66107fe448d921237d58fefacd6: Neither blocks nor log segments found. Creating new log.
I20260924 07:07:32.357681 15665 tablet_bootstrap.cc:492] T 9aa912dd467e49869dac1ec178da50cb P ba2b320a6d3d4a9dace89b01a16ed1ac: Bootstrap starting.
I20260924 07:07:32.358018 15663 log.cc:824] T 9aa912dd467e49869dac1ec178da50cb P ba5ce66107fe448d921237d58fefacd6: Log is configured to *not* fsync() on all Append() calls
I20260924 07:07:32.358615 15664 tablet_bootstrap.cc:654] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53: Neither blocks nor log segments found. Creating new log.
I20260924 07:07:32.359694 15664 log.cc:824] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53: Log is configured to *not* fsync() on all Append() calls
I20260924 07:07:32.360288 15665 tablet_bootstrap.cc:654] T 9aa912dd467e49869dac1ec178da50cb P ba2b320a6d3d4a9dace89b01a16ed1ac: Neither blocks nor log segments found. Creating new log.
I20260924 07:07:32.361603 15665 log.cc:824] T 9aa912dd467e49869dac1ec178da50cb P ba2b320a6d3d4a9dace89b01a16ed1ac: Log is configured to *not* fsync() on all Append() calls
I20260924 07:07:32.362519 15664 tablet_bootstrap.cc:492] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53: No bootstrap required, opened a new log
I20260924 07:07:32.362690 15664 ts_tablet_manager.cc:1397] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260924 07:07:32.365093 15665 tablet_bootstrap.cc:492] T 9aa912dd467e49869dac1ec178da50cb P ba2b320a6d3d4a9dace89b01a16ed1ac: No bootstrap required, opened a new log
I20260924 07:07:32.365258 15665 ts_tablet_manager.cc:1397] T 9aa912dd467e49869dac1ec178da50cb P ba2b320a6d3d4a9dace89b01a16ed1ac: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 07:07:32.366291 15664 raft_consensus.cc:348] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5ce66107fe448d921237d58fefacd6" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33667 } } peers { permanent_uuid: "ba2b320a6d3d4a9dace89b01a16ed1ac" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 37977 } } peers { permanent_uuid: "013c9e2171f34b3fb438e6a80f7c9f53" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 40571 } }
I20260924 07:07:32.366533 15664 raft_consensus.cc:374] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:07:32.366576 15664 raft_consensus.cc:729] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 013c9e2171f34b3fb438e6a80f7c9f53, State: Initialized, Role: FOLLOWER
I20260924 07:07:32.366997 15664 consensus_queue.cc:260] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5ce66107fe448d921237d58fefacd6" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33667 } } peers { permanent_uuid: "ba2b320a6d3d4a9dace89b01a16ed1ac" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 37977 } } peers { permanent_uuid: "013c9e2171f34b3fb438e6a80f7c9f53" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 40571 } }
I20260924 07:07:32.368151 15664 ts_tablet_manager.cc:1428] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 07:07:32.368534 15645 heartbeater.cc:503] Master 127.3.217.62:33039 was elected leader, sending a full tablet report...
I20260924 07:07:32.369931 15665 raft_consensus.cc:348] T 9aa912dd467e49869dac1ec178da50cb P ba2b320a6d3d4a9dace89b01a16ed1ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5ce66107fe448d921237d58fefacd6" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33667 } } peers { permanent_uuid: "ba2b320a6d3d4a9dace89b01a16ed1ac" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 37977 } } peers { permanent_uuid: "013c9e2171f34b3fb438e6a80f7c9f53" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 40571 } }
I20260924 07:07:32.370225 15665 raft_consensus.cc:374] T 9aa912dd467e49869dac1ec178da50cb P ba2b320a6d3d4a9dace89b01a16ed1ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:07:32.370311 15665 raft_consensus.cc:729] T 9aa912dd467e49869dac1ec178da50cb P ba2b320a6d3d4a9dace89b01a16ed1ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba2b320a6d3d4a9dace89b01a16ed1ac, State: Initialized, Role: FOLLOWER
I20260924 07:07:32.370807 15665 consensus_queue.cc:260] T 9aa912dd467e49869dac1ec178da50cb P ba2b320a6d3d4a9dace89b01a16ed1ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5ce66107fe448d921237d58fefacd6" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33667 } } peers { permanent_uuid: "ba2b320a6d3d4a9dace89b01a16ed1ac" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 37977 } } peers { permanent_uuid: "013c9e2171f34b3fb438e6a80f7c9f53" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 40571 } }
I20260924 07:07:32.371760 15665 ts_tablet_manager.cc:1428] T 9aa912dd467e49869dac1ec178da50cb P ba2b320a6d3d4a9dace89b01a16ed1ac: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:07:32.372113 15511 heartbeater.cc:503] Master 127.3.217.62:33039 was elected leader, sending a full tablet report...
I20260924 07:07:32.375970 15669 raft_consensus.cc:482] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:07:32.376155 15669 raft_consensus.cc:504] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5ce66107fe448d921237d58fefacd6" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33667 } } peers { permanent_uuid: "ba2b320a6d3d4a9dace89b01a16ed1ac" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 37977 } } peers { permanent_uuid: "013c9e2171f34b3fb438e6a80f7c9f53" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 40571 } }
I20260924 07:07:32.382469 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:07:32.377411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:33667 (local address 127.3.217.3:39059)
0924 07:07:32.377633 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 07:07:32.377648 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:07:32.377728 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:07:32.378331 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:07:32.378334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:07:32.378351 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:07:32.378451 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:07:32.378456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.379877 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.379880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:07:32.381099 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:07:32.381109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.382071 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.382075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:07:32.382174 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:07:32.382196 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:07:32.382268 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 07:07:32.382344 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260924 07:07:32.383065 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:07:32.377892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:39059 (local address 127.3.217.1:33667)
0924 07:07:32.378010 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 07:07:32.378013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:07:32.378025 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:07:32.378070 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:07:32.378073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:07:32.378117 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:07:32.378187 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:07:32.378566 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.379367 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.381236 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.381851 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.381948 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:07:32.382009 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 07:07:32.382876 (+ 867us) server_negotiation.cc:299] Negotiation successful
0924 07:07:32.382954 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:07:32.383659 15327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa912dd467e49869dac1ec178da50cb" candidate_uuid: "013c9e2171f34b3fb438e6a80f7c9f53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba5ce66107fe448d921237d58fefacd6" is_pre_election: true
W20260924 07:07:32.386360 15535 leader_election.cc:341] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ba5ce66107fe448d921237d58fefacd6 (127.3.217.1:33667): Illegal state: must be running to vote when last-logged opid is not known
I20260924 07:07:32.391901 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:07:32.386896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:37977 (local address 127.3.217.3:37909)
0924 07:07:32.387312 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0924 07:07:32.387325 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:07:32.387406 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:07:32.387732 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:07:32.387735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:07:32.387750 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:07:32.387899 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:07:32.387908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.389273 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.389276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:07:32.390469 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:07:32.390480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.391519 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.391523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:07:32.391609 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:07:32.391631 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:07:32.391706 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 07:07:32.391767 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353}
I20260924 07:07:32.392340 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:07:32.387119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:37909 (local address 127.3.217.2:37977)
0924 07:07:32.387231 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 07:07:32.387235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:07:32.387477 (+ 242us) server_negotiation.cc:407] Connection header received
0924 07:07:32.387541 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:07:32.387544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:07:32.387586 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:07:32.387642 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:07:32.388047 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.388946 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.390611 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.391226 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.391331 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:07:32.391395 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 07:07:32.392133 (+ 738us) server_negotiation.cc:299] Negotiation successful
0924 07:07:32.392208 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 07:07:32.392598 15512 tablet.cc:2369] T 9aa912dd467e49869dac1ec178da50cb P ba2b320a6d3d4a9dace89b01a16ed1ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:07:32.393761 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa912dd467e49869dac1ec178da50cb" candidate_uuid: "013c9e2171f34b3fb438e6a80f7c9f53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba2b320a6d3d4a9dace89b01a16ed1ac" is_pre_election: true
I20260924 07:07:32.394039 15462 raft_consensus.cc:2454] T 9aa912dd467e49869dac1ec178da50cb P ba2b320a6d3d4a9dace89b01a16ed1ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 013c9e2171f34b3fb438e6a80f7c9f53 in term 0.
I20260924 07:07:32.394507 15535 leader_election.cc:302] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 013c9e2171f34b3fb438e6a80f7c9f53, ba2b320a6d3d4a9dace89b01a16ed1ac; no voters: ba5ce66107fe448d921237d58fefacd6
I20260924 07:07:32.395004 15669 leader_election.cc:288] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba5ce66107fe448d921237d58fefacd6 (127.3.217.1:33667), ba2b320a6d3d4a9dace89b01a16ed1ac (127.3.217.2:37977)
I20260924 07:07:32.395345 15669 raft_consensus.cc:2781] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:07:32.395463 15669 raft_consensus.cc:482] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:07:32.395566 15669 raft_consensus.cc:3037] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:07:32.397030 15669 raft_consensus.cc:504] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5ce66107fe448d921237d58fefacd6" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33667 } } peers { permanent_uuid: "ba2b320a6d3d4a9dace89b01a16ed1ac" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 37977 } } peers { permanent_uuid: "013c9e2171f34b3fb438e6a80f7c9f53" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 40571 } }
I20260924 07:07:32.398049 15327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa912dd467e49869dac1ec178da50cb" candidate_uuid: "013c9e2171f34b3fb438e6a80f7c9f53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba5ce66107fe448d921237d58fefacd6"
W20260924 07:07:32.398517 15535 leader_election.cc:341] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ba5ce66107fe448d921237d58fefacd6 (127.3.217.1:33667): Illegal state: must be running to vote when last-logged opid is not known
I20260924 07:07:32.399013 15663 tablet_bootstrap.cc:492] T 9aa912dd467e49869dac1ec178da50cb P ba5ce66107fe448d921237d58fefacd6: No bootstrap required, opened a new log
I20260924 07:07:32.399005 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa912dd467e49869dac1ec178da50cb" candidate_uuid: "013c9e2171f34b3fb438e6a80f7c9f53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba2b320a6d3d4a9dace89b01a16ed1ac"
I20260924 07:07:32.399112 15462 raft_consensus.cc:3037] T 9aa912dd467e49869dac1ec178da50cb P ba2b320a6d3d4a9dace89b01a16ed1ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:07:32.399226 15663 ts_tablet_manager.cc:1397] T 9aa912dd467e49869dac1ec178da50cb P ba5ce66107fe448d921237d58fefacd6: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.000s
I20260924 07:07:32.400378 15462 raft_consensus.cc:2454] T 9aa912dd467e49869dac1ec178da50cb P ba2b320a6d3d4a9dace89b01a16ed1ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 013c9e2171f34b3fb438e6a80f7c9f53 in term 1.
I20260924 07:07:32.400754 15535 leader_election.cc:302] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 013c9e2171f34b3fb438e6a80f7c9f53, ba2b320a6d3d4a9dace89b01a16ed1ac; no voters: ba5ce66107fe448d921237d58fefacd6
I20260924 07:07:32.400931 15669 leader_election.cc:288] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53 [CANDIDATE]: Term 1 election: Requested vote from peers ba5ce66107fe448d921237d58fefacd6 (127.3.217.1:33667), ba2b320a6d3d4a9dace89b01a16ed1ac (127.3.217.2:37977)
I20260924 07:07:32.401098 15669 raft_consensus.cc:2781] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:07:32.401217 15669 raft_consensus.cc:686] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53 [term 1 LEADER]: Becoming Leader. State: Replica: 013c9e2171f34b3fb438e6a80f7c9f53, State: Running, Role: LEADER
I20260924 07:07:32.401535 15669 consensus_queue.cc:237] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5ce66107fe448d921237d58fefacd6" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33667 } } peers { permanent_uuid: "ba2b320a6d3d4a9dace89b01a16ed1ac" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 37977 } } peers { permanent_uuid: "013c9e2171f34b3fb438e6a80f7c9f53" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 40571 } }
I20260924 07:07:32.402225 15663 raft_consensus.cc:348] T 9aa912dd467e49869dac1ec178da50cb P ba5ce66107fe448d921237d58fefacd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5ce66107fe448d921237d58fefacd6" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33667 } } peers { permanent_uuid: "ba2b320a6d3d4a9dace89b01a16ed1ac" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 37977 } } peers { permanent_uuid: "013c9e2171f34b3fb438e6a80f7c9f53" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 40571 } }
I20260924 07:07:32.402454 15663 raft_consensus.cc:374] T 9aa912dd467e49869dac1ec178da50cb P ba5ce66107fe448d921237d58fefacd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:07:32.402506 15663 raft_consensus.cc:729] T 9aa912dd467e49869dac1ec178da50cb P ba5ce66107fe448d921237d58fefacd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba5ce66107fe448d921237d58fefacd6, State: Initialized, Role: FOLLOWER
I20260924 07:07:32.402966 15663 consensus_queue.cc:260] T 9aa912dd467e49869dac1ec178da50cb P ba5ce66107fe448d921237d58fefacd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5ce66107fe448d921237d58fefacd6" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33667 } } peers { permanent_uuid: "ba2b320a6d3d4a9dace89b01a16ed1ac" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 37977 } } peers { permanent_uuid: "013c9e2171f34b3fb438e6a80f7c9f53" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 40571 } }
I20260924 07:07:32.404028 15663 ts_tablet_manager.cc:1428] T 9aa912dd467e49869dac1ec178da50cb P ba5ce66107fe448d921237d58fefacd6: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:07:32.404460 15373 heartbeater.cc:503] Master 127.3.217.62:33039 was elected leader, sending a full tablet report...
I20260924 07:07:32.404904 15160 catalog_manager.cc:5495] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 013c9e2171f34b3fb438e6a80f7c9f53 (127.3.217.3). New cstate: current_term: 1 leader_uuid: "013c9e2171f34b3fb438e6a80f7c9f53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5ce66107fe448d921237d58fefacd6" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba2b320a6d3d4a9dace89b01a16ed1ac" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 37977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "013c9e2171f34b3fb438e6a80f7c9f53" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 40571 } health_report { overall_health: HEALTHY } } }
I20260924 07:07:32.419027 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:07:32.414520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50134 (local address 127.3.217.3:40571)
0924 07:07:32.414622 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 07:07:32.414626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:07:32.414638 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:07:32.414699 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:07:32.414702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:07:32.414759 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:07:32.414868 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:07:32.415238 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.415952 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.417077 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:07:32.417383 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:07:32.417464 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:07:32.417840 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:07:32.418315 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:07:32.418407 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 07:07:32.418835 (+ 428us) server_negotiation.cc:299] Negotiation successful
0924 07:07:32.418888 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:07:32.427659 15462 raft_consensus.cc:1261] T 9aa912dd467e49869dac1ec178da50cb P ba2b320a6d3d4a9dace89b01a16ed1ac [term 1 FOLLOWER]: Refusing update from remote peer 013c9e2171f34b3fb438e6a80f7c9f53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:07:32.428397 15327 raft_consensus.cc:3037] T 9aa912dd467e49869dac1ec178da50cb P ba5ce66107fe448d921237d58fefacd6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:07:32.430227 15327 raft_consensus.cc:1261] T 9aa912dd467e49869dac1ec178da50cb P ba5ce66107fe448d921237d58fefacd6 [term 1 FOLLOWER]: Refusing update from remote peer 013c9e2171f34b3fb438e6a80f7c9f53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:07:32.431744 15671 consensus_queue.cc:1035] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba5ce66107fe448d921237d58fefacd6" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 33667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:07:32.432016 15669 consensus_queue.cc:1035] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba2b320a6d3d4a9dace89b01a16ed1ac" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 37977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:07:32.442639 15675 mvcc.cc:205] Tried to move back new op lower bound from 7332797040334372864 to 7332797040239808512. Current Snapshot: MvccSnapshot[applied={T|T < 7332797040334372864}]
I20260924 07:07:32.444831 15676 mvcc.cc:205] Tried to move back new op lower bound from 7332797040334372864 to 7332797040239808512. Current Snapshot: MvccSnapshot[applied={T|T < 7332797040334372864}]
I20260924 07:07:32.449906 15677 mvcc.cc:205] Tried to move back new op lower bound from 7332797040334372864 to 7332797040239808512. Current Snapshot: MvccSnapshot[applied={T|T < 7332797040334372864}]
I20260924 07:07:32.453065 15159 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:07:32.453222 15159 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:07:32.455744 15159 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e6325ad640a4f61b25d6d5ed8b76ff7: Sending DeleteTablet for 3 replicas of tablet 9aa912dd467e49869dac1ec178da50cb
I20260924 07:07:32.456743 15307 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa912dd467e49869dac1ec178da50cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:07:32 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:54052
I20260924 07:07:32.456842 15445 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa912dd467e49869dac1ec178da50cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:07:32 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:39856
I20260924 07:07:32.457548 15575 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa912dd467e49869dac1ec178da50cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:07:32 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:50128
I20260924 07:07:32.457793 15685 tablet_replica.cc:330] T 9aa912dd467e49869dac1ec178da50cb P ba2b320a6d3d4a9dace89b01a16ed1ac: stopping tablet replica
I20260924 07:07:32.458079 15685 raft_consensus.cc:2229] T 9aa912dd467e49869dac1ec178da50cb P ba2b320a6d3d4a9dace89b01a16ed1ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:07:32.458393 15685 raft_consensus.cc:2258] T 9aa912dd467e49869dac1ec178da50cb P ba2b320a6d3d4a9dace89b01a16ed1ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:07:32.458485 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 15242
I20260924 07:07:32.461067 15685 ts_tablet_manager.cc:1905] T 9aa912dd467e49869dac1ec178da50cb P ba2b320a6d3d4a9dace89b01a16ed1ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:07:32.463683 15686 tablet_replica.cc:330] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53: stopping tablet replica
I20260924 07:07:32.464007 15685 ts_tablet_manager.cc:1918] T 9aa912dd467e49869dac1ec178da50cb P ba2b320a6d3d4a9dace89b01a16ed1ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:07:32.464078 15686 raft_consensus.cc:2229] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:07:32.464164 15685 log.cc:1196] T 9aa912dd467e49869dac1ec178da50cb P ba2b320a6d3d4a9dace89b01a16ed1ac: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/9aa912dd467e49869dac1ec178da50cb
I20260924 07:07:32.464486 15686 raft_consensus.cc:2258] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:07:32.464519 15685 ts_tablet_manager.cc:1939] T 9aa912dd467e49869dac1ec178da50cb P ba2b320a6d3d4a9dace89b01a16ed1ac: Deleting consensus metadata
I20260924 07:07:32.465652 15148 catalog_manager.cc:4841] TS ba2b320a6d3d4a9dace89b01a16ed1ac (127.3.217.2:37977): tablet 9aa912dd467e49869dac1ec178da50cb (table test-table [id=e1fddfd2dcde497a85cbb7d13a856e8f]) successfully deleted
I20260924 07:07:32.468742 15686 ts_tablet_manager.cc:1905] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 07:07:32.468809 15148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.217.1:33667 (error 108)
I20260924 07:07:32.469647 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 15381
I20260924 07:07:32.471767 15686 ts_tablet_manager.cc:1918] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:07:32.471892 15686 log.cc:1196] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/9aa912dd467e49869dac1ec178da50cb
I20260924 07:07:32.472220 15686 ts_tablet_manager.cc:1939] T 9aa912dd467e49869dac1ec178da50cb P 013c9e2171f34b3fb438e6a80f7c9f53: Deleting consensus metadata
I20260924 07:07:32.473416 15147 catalog_manager.cc:4841] TS 013c9e2171f34b3fb438e6a80f7c9f53 (127.3.217.3:40571): tablet 9aa912dd467e49869dac1ec178da50cb (table test-table [id=e1fddfd2dcde497a85cbb7d13a856e8f]) successfully deleted
I20260924 07:07:32.477344 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 15516
I20260924 07:07:32.479343 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:07:32.479028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:33667 (local address 127.0.0.1:54062)
0924 07:07:32.479188 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 07:07:32.479247 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.1:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260924 07:07:32.479789 15148 catalog_manager.cc:4584] TS ba5ce66107fe448d921237d58fefacd6 (127.3.217.1:33667): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9aa912dd467e49869dac1ec178da50cb: Network error: Client connection negotiation failed: client connection to 127.3.217.1:33667: connect: Connection refused (error 111)
I20260924 07:07:32.483495 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 15129
I20260924 07:07:32.489496 3940 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 07:07:32.823735 3940 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 07:07:32.823859 3940 mini_ranger.cc:66] Stopping Ranger...
I20260924 07:07:33.158713 3940 mini_ranger.cc:68] Stopped Ranger
2026-09-24 07:07:33.159 UTC [14714] LOG: received smart shutdown request
2026-09-24 07:07:33.162 UTC [14714] LOG: background worker "logical replication launcher" (PID 14723) exited with exit code 1
2026-09-24 07:07:33.163 UTC [14718] LOG: shutting down
2026-09-24 07:07:33.362 UTC [14714] LOG: database system is shut down
2026-09-24T07:07:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47981 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:07:33 dist-test-slave-2p7j krb5kdc[15696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:07:33 dist-test-slave-2p7j krb5kdc[15696](info): set up 2 sockets
Sep 24 07:07:33 dist-test-slave-2p7j krb5kdc[15696](info): commencing operation
krb5kdc: starting...
W20260924 07:07:34.436354 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:07:34 dist-test-slave-2p7j krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:07:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:07:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.217.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.217.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.217.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/rangeradmin_127.3.217.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.217.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/rangeradmin_127.3.217.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.217.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.217.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.217.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/rangerlookup_127.3.217.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.217.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/rangerlookup_127.3.217.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.217.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.217.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.217.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/HTTP_127.3.217.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.217.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/HTTP_127.3.217.20@KRBTEST.COM.keytab.
I20260924 07:07:34.597975 3940 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskSXrqy4/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/postgres -l logfile start
2026-09-24 07:07:37.158 UTC [15726] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 07:07:37.158 UTC [15726] LOG: listening on IPv4 address "127.3.217.20", port 40145
2026-09-24 07:07:37.162 UTC [15726] LOG: listening on Unix socket "/tmp/.s.PGSQL.40145"
2026-09-24 07:07:37.190 UTC [15729] LOG: database system was shut down at 2026-09-24 07:07:35 UTC
2026-09-24 07:07:37.194 UTC [15726] LOG: database system is ready to accept connections
I20260924 07:07:38.075819 3940 mini_postgres.cc:96] Postgres bound to 40145
2026-09-24 07:07:38.082 UTC [15737] FATAL: database "slave" does not exist
127.3.217.20:40145 - accepting connections
I20260924 07:07:38.083141 3940 mini_ranger.cc:161] Starting Ranger...
I20260924 07:07:38.098068 3940 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 07:07:38.689026 3940 mini_ranger.cc:87] Created ranger Postgres database
I20260924 07:07:38.689103 3940 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ranger-admin
2026-09-24 07:07:38,962 [I] DB FLAVOR :POSTGRES
2026-09-24 07:07:38,962 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 07:07:38,963 [I] Checking connection..
2026-09-24 07:07:38,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:40145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 07:07:39,199 [I] Checking connection passed.
2026-09-24 07:07:39,200 [I] --------- Verifying version history table ---------
2026-09-24 07:07:39,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:40145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:07:39,453 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 07:07:39,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:40145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:07:39,714 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 07:07:39,715 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 07:07:39,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:40145/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 07:07:39.967 UTC [15801] WARNING: there is no transaction in progress
2026-09-24 07:07:39,973 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 07:07:39,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:40145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:07:40,219 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 07:07:40,219 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 07:07:40,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:40145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 07:07:40,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:40145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 07:07:40,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:40145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2p7j.c.gcp-upstream.internal','N') ;"
2026-09-24 07:07:40,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:40145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 07:07:41,201 [I] Table x_portal_user does not exist in database ranger
2026-09-24 07:07:41,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:40145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 07:07:41,458 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 07:07:41,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:40145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 07:07:41,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:40145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 07:07:41,997 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 07:07:41,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:40145/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 07:07:43.071 UTC [15909] WARNING: there is no transaction in progress
2026-09-24 07:07:43.088 UTC [15909] WARNING: there is no transaction in progress
2026-09-24 07:07:43.106 UTC [15909] WARNING: there is no transaction in progress
2026-09-24 07:07:43.123 UTC [15909] WARNING: there is no transaction in progress
2026-09-24 07:07:43.142 UTC [15909] WARNING: there is no transaction in progress
2026-09-24 07:07:43.162 UTC [15909] WARNING: there is no transaction in progress
2026-09-24 07:07:43.226 UTC [15909] WARNING: there is no transaction in progress
2026-09-24 07:07:43.233 UTC [15909] WARNING: there is no transaction in progress
2026-09-24 07:07:43.251 UTC [15909] WARNING: there is no transaction in progress
2026-09-24 07:07:43.261 UTC [15909] WARNING: there is no transaction in progress
2026-09-24 07:07:43.272 UTC [15909] WARNING: there is no transaction in progress
2026-09-24 07:07:43.287 UTC [15909] WARNING: there is no transaction in progress
2026-09-24 07:07:43.298 UTC [15909] WARNING: there is no transaction in progress
2026-09-24 07:07:43.305 UTC [15909] WARNING: there is no transaction in progress
2026-09-24 07:07:43.313 UTC [15909] WARNING: there is no transaction in progress
2026-09-24 07:07:43.322 UTC [15909] WARNING: there is no transaction in progress
2026-09-24 07:07:44,001 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 07:07:44,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:40145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 07:07:44,243 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 07:07:44,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:40145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 07:07:44,478 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 07:07:44,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:40145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 07:07:44,745 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 07:07:44,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:40145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 07:07:44,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:40145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 07:07:45,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:40145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2p7j.c.gcp-upstream.internal';"
2026-09-24 07:07:45,479 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 07:07:45,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:40145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 07:07:45,720 [I] DB_PATCHES have already been applied
I20260924 07:07:45.726078 3940 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ranger-admin:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskSXrqy4/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 07:07:45.726140 3940 mini_ranger.cc:193] Using host: 127.3.217.20
I20260924 07:07:45.729621 3940 mini_ranger.cc:227] Ranger admin URL: http://127.3.217.20:37503
Sep 24, 2026 7:07:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:07:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:07:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 7:07:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 7:07:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 7:07:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 7:07:46 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 07:07:46.737780 3940 mini_ranger.cc:160] Time spent starting Ranger: real 8.655s user 0.000s sys 0.009s
Sep 24, 2026 7:08:06 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 7:08:06 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 7:08:06 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 7:08:06 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 7:08:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 7:08:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 7:08:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 7:08:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 7:08:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 7:08:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 7:08:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 7:08:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 7:08:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 7:08:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 7:08:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 7:08:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 7:08:06 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 7:08:08 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 07:08:08.612653 3940 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.217.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.217.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.217.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/rangerkms_127.3.217.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.217.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/rangerkms_127.3.217.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.217.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.217.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.217.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/HTTP_127.3.217.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.217.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/HTTP_127.3.217.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 07:08:08 dist-test-slave-2p7j krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233688, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 07:08:08.724251 3940 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 07:08:08.734978 3940 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 07:08:09.968101 3940 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 07:08:09.968179 3940 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ranger-kms
I20260924 07:08:09.968199 3940 mini_ranger_kms.cc:227] Using postgres at 127.3.217.20:40145
2026-09-24 07:08:10,145 [I] DB FLAVOR :POSTGRES
2026-09-24 07:08:10,145 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 07:08:10,145 [I] Checking connection
2026-09-24 07:08:10,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:40145/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 07:08:10,415 [I] connection success
2026-09-24 07:08:10,415 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 07:08:10,415 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 07:08:10,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:40145/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 07:08:10,698 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 07:08:10,698 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 07:08:10,698 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 07:08:10,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.217.20:40145/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 07:08:11,014 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 07:08:11.630630 3940 mini_ranger_kms.cc:310] Created kms service
I20260924 07:08:11.867374 3940 mini_ranger_kms.cc:326] Created kudu user
I20260924 07:08:11.916448 3940 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 07:08:11 dist-test-slave-2p7j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233688, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.217.20@KRBTEST.COM
I20260924 07:08:12.352633 3940 mini_ranger_kms.cc:384] Added ranger policy
I20260924 07:08:12.352802 3940 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ranger-kms:/tmp/dist-test-taskSXrqy4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskSXrqy4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 07:08:12.352824 3940 mini_ranger_kms.cc:240] Using host: 127.3.217.20
I20260924 07:08:12.358043 3940 mini_ranger_kms.cc:276] Ranger KMS PID: 16089
I20260924 07:08:12.358129 3940 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.217.20:39933
Sep 24, 2026 7:08:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:08:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:08:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 7:08:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 7:08:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 7:08:13 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 07:08:14.488590 3940 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.764s user 0.018s sys 0.142s
I20260924 07:08:14.488755 3940 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 07:08:14.488842 3940 mini_ranger_kms.cc:401] 127.3.217.20:39933/kms/v1/keys
Sep 24 07:08:14 dist-test-slave-2p7j krb5kdc[15696](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790233694, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.217.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 07:08:15 dist-test-slave-2p7j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233694, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.217.20@KRBTEST.COM for HTTP/127.3.217.20@KRBTEST.COM
Sep 24 07:08:15 dist-test-slave-2p7j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233694, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.217.20@KRBTEST.COM for HTTP/127.3.217.20@KRBTEST.COM
Sep 24 07:08:18 dist-test-slave-2p7j krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:08:18.061883 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:38767
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:44009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:38767
--ranger_config_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.217.20:39933/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:08:18.197822 16140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:18.198441 16140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:18.198546 16140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:18.204209 16140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:08:18.204298 16140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:18.204327 16140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:18.204352 16140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:08:18.204372 16140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:08:18.213605 16140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:44009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.217.20:39933/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:38767
--ranger_config_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:38767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:18.215209 16140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:18.216414 16140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:18.229615 16146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:18.229889 16145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:18.229889 16140 server_base.cc:1034] running on GCE node
W20260924 07:08:18.230436 16148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:18.231489 16140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:18.233011 16140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:18.234457 16140 hybrid_clock.cc:648] HybridClock initialized: now 1790233698234423 us; error 36 us; skew 500 ppm
Sep 24 07:08:18 dist-test-slave-2p7j krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233698, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:18.239115 16140 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:08:18.240531 16140 webserver.cc:460] Webserver started at http://127.3.217.62:39545/ using document root <none> and password file <none>
I20260924 07:08:18.241097 16140 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:18.241158 16140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:08:18.241351 16140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:08:18 dist-test-slave-2p7j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233698, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for HTTP/127.3.217.20@KRBTEST.COM
I20260924 07:08:18.305176 16140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "e3bfbf3b0cdc4e52b8463453d0406773"
format_stamp: "Formatted at 2026-09-24 07:08:18 on dist-test-slave-2p7j"
server_key: "4bb3b1ae5f10504d2f84e66d058f317a"
server_key_iv: "6e607f1f520af632b318fe99aa4d9873"
server_key_version: "kuduclusterkey@0"
I20260924 07:08:18.306258 16140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "e3bfbf3b0cdc4e52b8463453d0406773"
format_stamp: "Formatted at 2026-09-24 07:08:18 on dist-test-slave-2p7j"
server_key: "4bb3b1ae5f10504d2f84e66d058f317a"
server_key_iv: "6e607f1f520af632b318fe99aa4d9873"
server_key_version: "kuduclusterkey@0"
I20260924 07:08:18.311919 16140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 07:08:18.328397 16155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:18.330044 16140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 07:08:18.330188 16140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "e3bfbf3b0cdc4e52b8463453d0406773"
format_stamp: "Formatted at 2026-09-24 07:08:18 on dist-test-slave-2p7j"
server_key: "4bb3b1ae5f10504d2f84e66d058f317a"
server_key_iv: "6e607f1f520af632b318fe99aa4d9873"
server_key_version: "kuduclusterkey@0"
I20260924 07:08:18.330632 16140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:18.347623 16140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:18.351743 16140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:18.351961 16140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:18.361304 16140 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:38767
I20260924 07:08:18.361310 16207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:38767 every 8 connection(s)
I20260924 07:08:18.362284 16140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:08:18.365047 16208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:08:18.370497 16208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773: Bootstrap starting.
I20260924 07:08:18.372030 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 16140
I20260924 07:08:18.372148 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:08:18.372473 3940 external_mini_cluster.cc:1428] Setting key 61999b84753a7a6705aecc472fa51b50
I20260924 07:08:18.373180 16208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773: Neither blocks nor log segments found. Creating new log.
I20260924 07:08:18.374436 16208 log.cc:824] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773: Log is configured to *not* fsync() on all Append() calls
I20260924 07:08:18.376520 16208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773: No bootstrap required, opened a new log
I20260924 07:08:18.380901 16208 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bfbf3b0cdc4e52b8463453d0406773" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 38767 } }
I20260924 07:08:18.381278 16208 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:08:18.381359 16208 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3bfbf3b0cdc4e52b8463453d0406773, State: Initialized, Role: FOLLOWER
I20260924 07:08:18.381824 16208 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bfbf3b0cdc4e52b8463453d0406773" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 38767 } }
I20260924 07:08:18.381989 16208 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:08:18.382086 16208 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:08:18.382194 16208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 07:08:18 dist-test-slave-2p7j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:08:18.384370 16208 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bfbf3b0cdc4e52b8463453d0406773" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 38767 } }
I20260924 07:08:18.384943 16208 leader_election.cc:302] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3bfbf3b0cdc4e52b8463453d0406773; no voters:
I20260924 07:08:18.385460 16208 leader_election.cc:288] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:08:18.385561 16213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:08:18.385845 16213 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [term 1 LEADER]: Becoming Leader. State: Replica: e3bfbf3b0cdc4e52b8463453d0406773, State: Running, Role: LEADER
I20260924 07:08:18.386227 16213 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bfbf3b0cdc4e52b8463453d0406773" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 38767 } }
I20260924 07:08:18.387136 16208 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:08:18.387619 16213 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3bfbf3b0cdc4e52b8463453d0406773. Latest consensus state: current_term: 1 leader_uuid: "e3bfbf3b0cdc4e52b8463453d0406773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bfbf3b0cdc4e52b8463453d0406773" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 38767 } } }
I20260924 07:08:18.387826 16213 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [sys.catalog]: This master's current role is: LEADER
I20260924 07:08:18.388093 16213 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3bfbf3b0cdc4e52b8463453d0406773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bfbf3b0cdc4e52b8463453d0406773" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 38767 } } }
I20260924 07:08:18.388239 16213 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [sys.catalog]: This master's current role is: LEADER
I20260924 07:08:18.389438 16208 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 07:08:18.393852 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:18.373717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52876 (local address 127.3.217.62:38767)
0924 07:08:18.374356 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:18.374375 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:18.374420 (+ 45us) server_negotiation.cc:407] Connection header received
0924 07:08:18.375293 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:18.375317 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:18.375815 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:18.376332 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:18.377185 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:18.378721 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:18.379998 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:18.380849 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:18.381048 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:18.385191 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:18.385198 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:18.385216 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:18.385250 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:18.387351 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:18.387925 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:18.387948 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:18.387954 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:18.388106 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:18.388433 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:18.388436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:18.388437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:18.388962 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:18.389825 (+ 863us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:18.390109 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 07:08:18.390399 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":131,"threads_started":1}
I20260924 07:08:18.393989 16218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:08:18.399982 16218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:08:18.407927 16218 catalog_manager.cc:1347] Generated new cluster ID: 4fc16afb26674ac0838bcd8deaff4480
I20260924 07:08:18.408002 16218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:08:18.434273 16218 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:08:18.436780 16218 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:08:18.446746 16218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773: Generated new TSK 0
I20260924 07:08:18.447854 16218 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
I20260924 07:08:19.220484 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.217.20:39933/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:38767
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:44009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:08:19.380112 16253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:19.380481 16253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:19.380568 16253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:19.387033 16253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:19.387163 16253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:19.387270 16253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:08:19.395002 16253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:44009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.217.20:39933/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:38767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:19.396494 16253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:19.397542 16253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:19.408900 16259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:19.409093 16253 server_base.cc:1034] running on GCE node
W20260924 07:08:19.409322 16261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:19.409361 16258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:19.409925 16253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:19.410596 16253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:19.411936 16253 hybrid_clock.cc:648] HybridClock initialized: now 1790233699411892 us; error 43 us; skew 500 ppm
Sep 24 07:08:19 dist-test-slave-2p7j krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233699, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:19.421391 16253 init.cc:377] Logged in from keytab as kudu/127.3.217.1@KRBTEST.COM (short username kudu)
I20260924 07:08:19.422775 16253 webserver.cc:460] Webserver started at http://127.3.217.1:38497/ using document root <none> and password file <none>
I20260924 07:08:19.423640 16253 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:19.423740 16253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:08:19.424108 16253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:08:19 dist-test-slave-2p7j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233699, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for HTTP/127.3.217.20@KRBTEST.COM
I20260924 07:08:19.445010 16253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "753452748bfb45188ab0f1b65c9be6de"
format_stamp: "Formatted at 2026-09-24 07:08:19 on dist-test-slave-2p7j"
server_key: "d8d1e0160e7eaf1373bc70289a15e0d6"
server_key_iv: "b1d56847282f321949b7c5f6de1a568e"
server_key_version: "kuduclusterkey@0"
I20260924 07:08:19.445734 16253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "753452748bfb45188ab0f1b65c9be6de"
format_stamp: "Formatted at 2026-09-24 07:08:19 on dist-test-slave-2p7j"
server_key: "d8d1e0160e7eaf1373bc70289a15e0d6"
server_key_iv: "b1d56847282f321949b7c5f6de1a568e"
server_key_version: "kuduclusterkey@0"
I20260924 07:08:19.455592 16253 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.002s sys 0.004s
I20260924 07:08:19.479223 16270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:19.481033 16253 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260924 07:08:19.481230 16253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "753452748bfb45188ab0f1b65c9be6de"
format_stamp: "Formatted at 2026-09-24 07:08:19 on dist-test-slave-2p7j"
server_key: "d8d1e0160e7eaf1373bc70289a15e0d6"
server_key_iv: "b1d56847282f321949b7c5f6de1a568e"
server_key_version: "kuduclusterkey@0"
I20260924 07:08:19.481441 16253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:19.498267 16253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:19.502808 16253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:19.503083 16253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:19.503871 16253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:08:19.505002 16253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:08:19.505102 16253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:19.505228 16253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:08:19.505301 16253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:19.527227 16253 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:36317
I20260924 07:08:19.529348 16253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:08:19.534497 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 16253
I20260924 07:08:19.534734 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:08:19.535128 3940 external_mini_cluster.cc:1428] Setting key f2fbca3c2454853959965a02b03fcafc
I20260924 07:08:19.532096 16383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:36317 every 8 connection(s)
Sep 24 07:08:19 dist-test-slave-2p7j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233699, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:08:19.558174 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:19.535397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:35901 (local address 127.3.217.62:38767)
0924 07:08:19.536148 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:19.536153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:19.536650 (+ 497us) server_negotiation.cc:407] Connection header received
0924 07:08:19.537841 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:19.537845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:19.537897 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:19.537976 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:19.539650 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:19.540521 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:19.542180 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:19.542537 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:19.542604 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:19.552455 (+ 9851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:19.552460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:19.552463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:19.552501 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:19.554692 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:19.555822 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:19.555827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:19.555830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:19.555967 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:19.556628 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:19.556633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:19.556635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:19.556912 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:19.557054 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:19.557826 (+ 772us) server_negotiation.cc:299] Negotiation successful
0924 07:08:19.557982 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":107,"threads_started":1}
I20260924 07:08:19.560151 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:19.534259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:38767 (local address 127.3.217.1:35901)
0924 07:08:19.536400 (+ 2141us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:19.536451 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:19.537515 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:19.538368 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:19.538378 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:19.538771 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:19.539458 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:19.539479 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:19.540672 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:19.540675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:19.542050 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:19.542060 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:19.542800 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:19.542803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:19.543003 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:19.543690 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:19.543710 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:19.552093 (+ 8383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:19.554903 (+ 2810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:19.554915 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:19.554945 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:19.555626 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:19.556102 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:19.556105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:19.556107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:19.556451 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:19.557143 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:19.557151 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:19.557527 (+ 376us) client_negotiation.cc:769] Sending connection context
0924 07:08:19.558660 (+ 1133us) client_negotiation.cc:240] Negotiation successful
0924 07:08:19.559043 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1933,"thread_start_us":141,"threads_started":1}
I20260924 07:08:19.561827 16384 heartbeater.cc:347] Connected to a master server at 127.3.217.62:38767
I20260924 07:08:19.562083 16384 heartbeater.cc:464] Registering TS with master...
I20260924 07:08:19.562669 16384 heartbeater.cc:511] Master 127.3.217.62:38767 requested a full tablet report, sending...
I20260924 07:08:19.564503 16172 ts_manager.cc:194] Registered new tserver with Master: 753452748bfb45188ab0f1b65c9be6de (127.3.217.1:36317)
WARNING: no policy specified for kudu/127.3.217.2@KRBTEST.COM; defaulting to no policy
I20260924 07:08:19.566123 16172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.1@KRBTEST.COM'} at 127.3.217.1:35901
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
I20260924 07:08:19.608287 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.217.20:39933/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:38767
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:44009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:08:19 dist-test-slave-2p7j krb5kdc[15696](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790233699, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 07:08:19.800989 16392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:19.801462 16392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:19.801560 16392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:19.808704 16392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:19.808917 16392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:19.809047 16392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:08:19.819993 16392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:44009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.217.20:39933/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:38767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:19.821434 16392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:19.822757 16392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:19.835628 16399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:19.836050 16398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:19.836300 16401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:19.837540 16392 server_base.cc:1034] running on GCE node
I20260924 07:08:19.838080 16392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:19.839172 16392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:19.840571 16392 hybrid_clock.cc:648] HybridClock initialized: now 1790233699840541 us; error 44 us; skew 500 ppm
Sep 24 07:08:19 dist-test-slave-2p7j krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233699, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:19.844108 16392 init.cc:377] Logged in from keytab as kudu/127.3.217.2@KRBTEST.COM (short username kudu)
I20260924 07:08:19.845208 16392 webserver.cc:460] Webserver started at http://127.3.217.2:36727/ using document root <none> and password file <none>
I20260924 07:08:19.845873 16392 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:19.845952 16392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:08:19.846161 16392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:08:19 dist-test-slave-2p7j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233699, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for HTTP/127.3.217.20@KRBTEST.COM
I20260924 07:08:19.871002 16392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "138b349fb78248f29c1fe0aa0ea86c7b"
format_stamp: "Formatted at 2026-09-24 07:08:19 on dist-test-slave-2p7j"
server_key: "a3ea1142364db1a18aee91c5ca4b6d1d"
server_key_iv: "fa6b8e4aee8eb9e67821e55d3550f424"
server_key_version: "kuduclusterkey@0"
I20260924 07:08:19.871914 16392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "138b349fb78248f29c1fe0aa0ea86c7b"
format_stamp: "Formatted at 2026-09-24 07:08:19 on dist-test-slave-2p7j"
server_key: "a3ea1142364db1a18aee91c5ca4b6d1d"
server_key_iv: "fa6b8e4aee8eb9e67821e55d3550f424"
server_key_version: "kuduclusterkey@0"
I20260924 07:08:19.877398 16392 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 07:08:19.918236 16408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:19.920153 16392 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.000s
I20260924 07:08:19.920323 16392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "138b349fb78248f29c1fe0aa0ea86c7b"
format_stamp: "Formatted at 2026-09-24 07:08:19 on dist-test-slave-2p7j"
server_key: "a3ea1142364db1a18aee91c5ca4b6d1d"
server_key_iv: "fa6b8e4aee8eb9e67821e55d3550f424"
server_key_version: "kuduclusterkey@0"
I20260924 07:08:19.920472 16392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:19.934144 16392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:19.937786 16392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:19.937979 16392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:19.938627 16392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:08:19.939489 16392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:08:19.939580 16392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:19.939663 16392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:08:19.939697 16392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:19.949457 16392 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:43945
I20260924 07:08:19.949491 16521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:43945 every 8 connection(s)
I20260924 07:08:19.950585 16392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:08:19.952268 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 16392
I20260924 07:08:19.952510 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:08:19.952826 3940 external_mini_cluster.cc:1428] Setting key 89c03b681c679b8ba0c4bbefe0614737
WARNING: no policy specified for kudu/127.3.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.3@KRBTEST.COM" created.
Sep 24 07:08:19 dist-test-slave-2p7j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233699, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:08:19.982290 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:19.954241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:45805 (local address 127.3.217.62:38767)
0924 07:08:19.954356 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:19.954360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:19.955074 (+ 714us) server_negotiation.cc:407] Connection header received
0924 07:08:19.964904 (+ 9830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:19.964908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:19.964970 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:19.965042 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:19.966717 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:19.967583 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:19.969494 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:19.969843 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:19.969925 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:19.977984 (+ 8059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:19.977989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:19.977992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:19.978029 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:19.979743 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:19.980444 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:19.980448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:19.980450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:19.980549 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:19.980978 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:19.980982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:19.980983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:19.981212 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:19.981294 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:19.981996 (+ 702us) server_negotiation.cc:299] Negotiation successful
0924 07:08:19.982133 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:08:19.984208 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:19.953075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:38767 (local address 127.3.217.2:45805)
0924 07:08:19.954939 (+ 1864us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:19.954990 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:19.964605 (+ 9615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:19.965481 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:19.965492 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:19.965847 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:19.966553 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:19.966573 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:19.967738 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:19.967741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:19.969345 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:19.969358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:19.970137 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:19.970141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:19.970347 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:19.975877 (+ 5530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:19.975909 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:19.977719 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:19.979902 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:19.979909 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:19.979929 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:19.980320 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:19.980655 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:19.980658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:19.980660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:19.980855 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:19.981347 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:19.981354 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:19.981726 (+ 372us) client_negotiation.cc:769] Sending connection context
0924 07:08:19.982755 (+ 1029us) client_negotiation.cc:240] Negotiation successful
0924 07:08:19.983113 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1654,"thread_start_us":171,"threads_started":1}
I20260924 07:08:19.985787 16522 heartbeater.cc:347] Connected to a master server at 127.3.217.62:38767
I20260924 07:08:19.986106 16522 heartbeater.cc:464] Registering TS with master...
I20260924 07:08:19.986724 16522 heartbeater.cc:511] Master 127.3.217.62:38767 requested a full tablet report, sending...
I20260924 07:08:19.987977 16172 ts_manager.cc:194] Registered new tserver with Master: 138b349fb78248f29c1fe0aa0ea86c7b (127.3.217.2:43945)
I20260924 07:08:19.988839 16172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.2@KRBTEST.COM'} at 127.3.217.2:45805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
I20260924 07:08:20.020206 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.217.20:39933/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:38767
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:44009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:08:20.193894 16527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:20.194281 16527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:20.194351 16527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Sep 24 07:08:20 dist-test-slave-2p7j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233699, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for HTTP/127.3.217.20@KRBTEST.COM
W20260924 07:08:20.201081 16527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:20.201157 16527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:20.201254 16527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:08:20.211581 16527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:44009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.217.20:39933/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:38767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:20.213212 16527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:20.214570 16527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:20.233804 16535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:20.234802 16533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:20.235216 16532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:20.236581 16527 server_base.cc:1034] running on GCE node
I20260924 07:08:20.237372 16527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:20.238188 16527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:20.241421 16527 hybrid_clock.cc:648] HybridClock initialized: now 1790233700241350 us; error 123 us; skew 500 ppm
Sep 24 07:08:20 dist-test-slave-2p7j krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233700, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:20.246052 16527 init.cc:377] Logged in from keytab as kudu/127.3.217.3@KRBTEST.COM (short username kudu)
I20260924 07:08:20.247171 16527 webserver.cc:460] Webserver started at http://127.3.217.3:37347/ using document root <none> and password file <none>
I20260924 07:08:20.247905 16527 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:20.247982 16527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:08:20.248194 16527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:08:20 dist-test-slave-2p7j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233700, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for HTTP/127.3.217.20@KRBTEST.COM
I20260924 07:08:20.277897 16527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "902c1f58a0134eb1b9c7f6fa938c2548"
format_stamp: "Formatted at 2026-09-24 07:08:20 on dist-test-slave-2p7j"
server_key: "54e8c2b9c758b0c8c5e890b890c16197"
server_key_iv: "cf23a25fb55f8f60de8acf4822e3d985"
server_key_version: "kuduclusterkey@0"
I20260924 07:08:20.279687 16527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "902c1f58a0134eb1b9c7f6fa938c2548"
format_stamp: "Formatted at 2026-09-24 07:08:20 on dist-test-slave-2p7j"
server_key: "54e8c2b9c758b0c8c5e890b890c16197"
server_key_iv: "cf23a25fb55f8f60de8acf4822e3d985"
server_key_version: "kuduclusterkey@0"
I20260924 07:08:20.313310 16527 fs_manager.cc:691] Time spent creating directory manager: real 0.026s user 0.002s sys 0.003s
I20260924 07:08:20.359968 16543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:20.362208 16527 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.001s
I20260924 07:08:20.362461 16527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "902c1f58a0134eb1b9c7f6fa938c2548"
format_stamp: "Formatted at 2026-09-24 07:08:20 on dist-test-slave-2p7j"
server_key: "54e8c2b9c758b0c8c5e890b890c16197"
server_key_iv: "cf23a25fb55f8f60de8acf4822e3d985"
server_key_version: "kuduclusterkey@0"
I20260924 07:08:20.362624 16527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:20.396162 16527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:20.401396 16527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:20.401758 16527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:20.402969 16527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:08:20.404306 16527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:08:20.404431 16527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:20.404538 16527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:08:20.404603 16527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:20.415599 16527 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:39585
I20260924 07:08:20.416918 16527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:08:20.421888 16656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:39585 every 8 connection(s)
I20260924 07:08:20.424448 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 16527
I20260924 07:08:20.424571 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:08:20.424878 3940 external_mini_cluster.cc:1428] Setting key 7ec2e893ed729ae2efc2ba92baeb4bbd
Sep 24 07:08:20 dist-test-slave-2p7j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233700, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:08:20.443477 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:20.423876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:49107 (local address 127.3.217.62:38767)
0924 07:08:20.424020 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:20.424024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:20.427637 (+ 3613us) server_negotiation.cc:407] Connection header received
0924 07:08:20.428877 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:20.428880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:20.428932 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:20.428979 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:20.430547 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:20.431359 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:20.432994 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:20.433318 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:20.433395 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:20.436493 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:20.436497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:20.436500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:20.436526 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:20.441061 (+ 4535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:20.441712 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:20.441716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:20.441718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:20.441805 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:20.442211 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:20.442214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:20.442216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:20.442439 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:20.442514 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:20.443186 (+ 672us) server_negotiation.cc:299] Negotiation successful
0924 07:08:20.443312 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 07:08:20.445322 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:20.424434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:38767 (local address 127.3.217.3:49107)
0924 07:08:20.427389 (+ 2955us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:20.427548 (+ 159us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:20.428637 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:20.429386 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:20.429395 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:20.429732 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:20.430386 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:20.430404 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:20.431532 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:20.431535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:20.432861 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:20.432872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:20.433609 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:20.433613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:20.433802 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:20.434499 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:20.434521 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:20.436306 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:20.441200 (+ 4894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:20.441207 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:20.441228 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:20.441595 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:20.441923 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:20.441926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:20.441928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:20.442110 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:20.442575 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:20.442582 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:20.442920 (+ 338us) client_negotiation.cc:769] Sending connection context
0924 07:08:20.443947 (+ 1027us) client_negotiation.cc:240] Negotiation successful
0924 07:08:20.444313 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2654,"thread_start_us":107,"threads_started":1}
I20260924 07:08:20.446722 16657 heartbeater.cc:347] Connected to a master server at 127.3.217.62:38767
I20260924 07:08:20.446973 16657 heartbeater.cc:464] Registering TS with master...
I20260924 07:08:20.447520 16657 heartbeater.cc:511] Master 127.3.217.62:38767 requested a full tablet report, sending...
I20260924 07:08:20.448616 16172 ts_manager.cc:194] Registered new tserver with Master: 902c1f58a0134eb1b9c7f6fa938c2548 (127.3.217.3:39585)
I20260924 07:08:20.449318 16172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.3@KRBTEST.COM'} at 127.3.217.3:49107
I20260924 07:08:20.451418 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:08:20.452301 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 16253
I20260924 07:08:20.465715 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 16392
I20260924 07:08:20.475683 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 16527
I20260924 07:08:20.483132 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 16140
I20260924 07:08:20.490540 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:38767
--webserver_interface=127.3.217.62
--webserver_port=39545
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:44009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:38767
--ranger_config_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.217.20:39933/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.217.20:39933/kms with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:08:20.611739 16660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:20.612020 16660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:20.612072 16660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:20.617885 16660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:08:20.617962 16660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:20.617982 16660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:20.617997 16660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:08:20.618011 16660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:08:20.624847 16660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:44009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.217.20:39933/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:38767
--ranger_config_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:38767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=39545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:20.625887 16660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:20.626840 16660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:20.635988 16665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:20.636058 16668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:20.636052 16660 server_base.cc:1034] running on GCE node
W20260924 07:08:20.636196 16666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:20.637130 16660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:20.638785 16660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:20.642176 16660 hybrid_clock.cc:648] HybridClock initialized: now 1790233700642190 us; error 92 us; skew 500 ppm
Sep 24 07:08:20 dist-test-slave-2p7j krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233700, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:20.645030 16660 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:08:20.645972 16660 webserver.cc:460] Webserver started at http://127.3.217.62:39545/ using document root <none> and password file <none>
I20260924 07:08:20.646526 16660 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:20.646602 16660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 07:08:20 dist-test-slave-2p7j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233700, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for HTTP/127.3.217.20@KRBTEST.COM
I20260924 07:08:20.717744 16660 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:08:20.720002 16676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:20.722607 16660 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:08:20.722764 16660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "e3bfbf3b0cdc4e52b8463453d0406773"
format_stamp: "Formatted at 2026-09-24 07:08:18 on dist-test-slave-2p7j"
server_key: "4bb3b1ae5f10504d2f84e66d058f317a"
server_key_iv: "6e607f1f520af632b318fe99aa4d9873"
server_key_version: "kuduclusterkey@0"
I20260924 07:08:20.725833 16660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:20.742214 16660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:20.750494 16660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:20.750752 16660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:20.759284 16660 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:38767
I20260924 07:08:20.759321 16728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:38767 every 8 connection(s)
I20260924 07:08:20.760433 16660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:08:20.760632 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 16660
I20260924 07:08:20.761744 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.217.20:39933/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:36317
--local_ip_for_outbound_sockets=127.3.217.1
--tserver_master_addrs=127.3.217.62:38767
--webserver_port=38497
--webserver_interface=127.3.217.1
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:44009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.217.20:39933/kms with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:08:20.772163 16729 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:08:20.775489 16729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773: Bootstrap starting.
I20260924 07:08:20.802356 16729 log.cc:824] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773: Log is configured to *not* fsync() on all Append() calls
I20260924 07:08:20.814493 16729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:08:20.814996 16729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773: Bootstrap complete.
I20260924 07:08:20.820801 16729 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bfbf3b0cdc4e52b8463453d0406773" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 38767 } }
I20260924 07:08:20.821424 16729 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3bfbf3b0cdc4e52b8463453d0406773, State: Initialized, Role: FOLLOWER
I20260924 07:08:20.822393 16729 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bfbf3b0cdc4e52b8463453d0406773" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 38767 } }
I20260924 07:08:20.822592 16729 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:08:20.822664 16729 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:08:20.822757 16729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:08:20.826138 16729 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bfbf3b0cdc4e52b8463453d0406773" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 38767 } }
I20260924 07:08:20.826979 16729 leader_election.cc:302] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3bfbf3b0cdc4e52b8463453d0406773; no voters:
I20260924 07:08:20.827371 16729 leader_election.cc:288] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 07:08:20.827783 16733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 07:08:20.828346 16733 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [term 2 LEADER]: Becoming Leader. State: Replica: e3bfbf3b0cdc4e52b8463453d0406773, State: Running, Role: LEADER
I20260924 07:08:20.828945 16733 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bfbf3b0cdc4e52b8463453d0406773" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 38767 } }
I20260924 07:08:20.829054 16729 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:08:20.833402 16729 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 07:08:20.834048 16733 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3bfbf3b0cdc4e52b8463453d0406773. Latest consensus state: current_term: 2 leader_uuid: "e3bfbf3b0cdc4e52b8463453d0406773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bfbf3b0cdc4e52b8463453d0406773" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 38767 } } }
I20260924 07:08:20.834197 16733 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [sys.catalog]: This master's current role is: LEADER
I20260924 07:08:20.838822 16735 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e3bfbf3b0cdc4e52b8463453d0406773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bfbf3b0cdc4e52b8463453d0406773" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 38767 } } }
I20260924 07:08:20.839112 16735 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773 [sys.catalog]: This master's current role is: LEADER
I20260924 07:08:20.838827 16747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:08:20.848971 16747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:08:20.849756 16747 catalog_manager.cc:1259] Loaded cluster ID: 4fc16afb26674ac0838bcd8deaff4480
I20260924 07:08:20.849864 16747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:08:20.852733 16747 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:08:20.853777 16747 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773: Loaded TSK: 0
I20260924 07:08:20.855257 16747 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:08:20.970340 16731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:20.970739 16731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:20.970813 16731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:20.979219 16731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:20.979315 16731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:20.979413 16731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:08:20.987847 16731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:44009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.217.20:39933/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:36317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=38497
--tserver_master_addrs=127.3.217.62:38767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:20.989220 16731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:20.990571 16731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:21.007333 16767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:21.007377 16769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:21.007815 16731 server_base.cc:1034] running on GCE node
W20260924 07:08:21.008073 16766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:21.008626 16731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:21.010036 16731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:21.013451 16731 hybrid_clock.cc:648] HybridClock initialized: now 1790233701013361 us; error 111 us; skew 500 ppm
Sep 24 07:08:21 dist-test-slave-2p7j krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233701, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:21.017421 16731 init.cc:377] Logged in from keytab as kudu/127.3.217.1@KRBTEST.COM (short username kudu)
I20260924 07:08:21.018630 16731 webserver.cc:460] Webserver started at http://127.3.217.1:38497/ using document root <none> and password file <none>
I20260924 07:08:21.019407 16731 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:21.019608 16731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 07:08:21 dist-test-slave-2p7j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233701, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for HTTP/127.3.217.20@KRBTEST.COM
I20260924 07:08:21.046881 16731 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260924 07:08:21.050711 16777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:21.052363 16731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:08:21.052496 16731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "753452748bfb45188ab0f1b65c9be6de"
format_stamp: "Formatted at 2026-09-24 07:08:19 on dist-test-slave-2p7j"
server_key: "d8d1e0160e7eaf1373bc70289a15e0d6"
server_key_iv: "b1d56847282f321949b7c5f6de1a568e"
server_key_version: "kuduclusterkey@0"
I20260924 07:08:21.053206 16731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:21.081278 16731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:21.087868 16731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:21.088361 16731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:21.089434 16731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:08:21.091183 16731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:08:21.091365 16731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:21.091439 16731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:08:21.091470 16731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:21.103873 16731 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:36317
I20260924 07:08:21.104233 16890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:36317 every 8 connection(s)
I20260924 07:08:21.105355 16731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:08:21.108669 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 16731
I20260924 07:08:21.109146 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.217.20:39933/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:43945
--local_ip_for_outbound_sockets=127.3.217.2
--tserver_master_addrs=127.3.217.62:38767
--webserver_port=36727
--webserver_interface=127.3.217.2
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:44009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.217.20:39933/kms with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:08:21 dist-test-slave-2p7j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233701, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:08:21.135560 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:21.108932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:38767 (local address 127.3.217.1:38125)
0924 07:08:21.112047 (+ 3115us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:21.112115 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:21.113188 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:21.114653 (+ 1465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:21.114661 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:21.115014 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:21.115689 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:21.115710 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:21.117586 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:21.117589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:21.118896 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:21.118908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:21.120110 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:21.120114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:21.120285 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:21.122511 (+ 2226us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:21.122544 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:21.128010 (+ 5466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:21.131540 (+ 3530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:21.131548 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:21.131569 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:21.132018 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:21.132492 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:21.132495 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:21.132497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:21.132735 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:21.133525 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:21.133531 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:21.133982 (+ 451us) client_negotiation.cc:769] Sending connection context
0924 07:08:21.134166 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 07:08:21.134503 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2885,"thread_start_us":139,"threads_started":1}
I20260924 07:08:21.137329 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:21.110137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:38125 (local address 127.3.217.62:38767)
0924 07:08:21.111594 (+ 1457us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:21.111620 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:21.113004 (+ 1384us) server_negotiation.cc:407] Connection header received
0924 07:08:21.113690 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:21.113707 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:21.114126 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:21.114452 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:21.115895 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:21.117431 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:21.119034 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:21.119748 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:21.119898 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:21.128415 (+ 8517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:21.128424 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:21.128447 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:21.128495 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:21.131119 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:21.132182 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:21.132190 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:21.132195 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:21.132379 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:21.132852 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:21.132855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:21.132857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:21.133344 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:21.133471 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:21.136184 (+ 2713us) server_negotiation.cc:299] Negotiation successful
0924 07:08:21.136481 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":116,"threads_started":1}
I20260924 07:08:21.140580 16891 heartbeater.cc:347] Connected to a master server at 127.3.217.62:38767
I20260924 07:08:21.141000 16891 heartbeater.cc:464] Registering TS with master...
I20260924 07:08:21.142470 16891 heartbeater.cc:511] Master 127.3.217.62:38767 requested a full tablet report, sending...
W20260924 07:08:21.146941 16891 heartbeater.cc:644] Failed to heartbeat to 127.3.217.62:38767 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 07:08:21.147187 16891 heartbeater.cc:464] Registering TS with master...
I20260924 07:08:21.147742 16891 heartbeater.cc:511] Master 127.3.217.62:38767 requested a full tablet report, sending...
I20260924 07:08:21.149374 16891 heartbeater.cc:464] Registering TS with master...
I20260924 07:08:21.149721 16891 heartbeater.cc:511] Master 127.3.217.62:38767 requested a full tablet report, sending...
W20260924 07:08:21.150591 16891 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:08:21.307884 16895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:21.308738 16895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:21.308887 16895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:21.319032 16895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:21.319224 16895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:21.319396 16895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:08:21.328646 16895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:44009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.217.20:39933/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:43945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=36727
--tserver_master_addrs=127.3.217.62:38767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:21.330415 16895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:21.332049 16895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:21.350032 16901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:21.350049 16900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:21.350049 16903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:21.350816 16895 server_base.cc:1034] running on GCE node
I20260924 07:08:21.351466 16895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:21.352283 16895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:21.353466 16895 hybrid_clock.cc:648] HybridClock initialized: now 1790233701353436 us; error 47 us; skew 500 ppm
Sep 24 07:08:21 dist-test-slave-2p7j krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233701, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:21.357025 16895 init.cc:377] Logged in from keytab as kudu/127.3.217.2@KRBTEST.COM (short username kudu)
I20260924 07:08:21.358072 16895 webserver.cc:460] Webserver started at http://127.3.217.2:36727/ using document root <none> and password file <none>
I20260924 07:08:21.358752 16895 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:21.358835 16895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 07:08:21 dist-test-slave-2p7j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233701, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for HTTP/127.3.217.20@KRBTEST.COM
I20260924 07:08:21.392030 16895 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.000s
I20260924 07:08:21.394685 16911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:21.396397 16895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:08:21.396533 16895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "138b349fb78248f29c1fe0aa0ea86c7b"
format_stamp: "Formatted at 2026-09-24 07:08:19 on dist-test-slave-2p7j"
server_key: "a3ea1142364db1a18aee91c5ca4b6d1d"
server_key_iv: "fa6b8e4aee8eb9e67821e55d3550f424"
server_key_version: "kuduclusterkey@0"
I20260924 07:08:21.397217 16895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:21.413832 16895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:21.417884 16895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:21.418124 16895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:21.418838 16895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:08:21.419871 16895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:08:21.419934 16895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:21.419981 16895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:08:21.420039 16895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:21.430119 16895 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:43945
I20260924 07:08:21.430387 17024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:43945 every 8 connection(s)
I20260924 07:08:21.431241 16895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:08:21.441396 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 16895
I20260924 07:08:21.441883 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.217.20:39933/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:39585
--local_ip_for_outbound_sockets=127.3.217.3
--tserver_master_addrs=127.3.217.62:38767
--webserver_port=37347
--webserver_interface=127.3.217.3
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:44009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.217.20:39933/kms with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:08:21 dist-test-slave-2p7j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233701, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:08:21.458653 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:21.437109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:60517 (local address 127.3.217.62:38767)
0924 07:08:21.437241 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:21.437246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:21.439938 (+ 2692us) server_negotiation.cc:407] Connection header received
0924 07:08:21.441647 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:21.441652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:21.441751 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:21.441811 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:21.443885 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:21.444720 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:21.446501 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:21.446810 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:21.446909 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:21.453172 (+ 6263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:21.453186 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:21.453195 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:21.453239 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:21.454914 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:21.456080 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:21.456086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:21.456088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:21.456224 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:21.456790 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:21.456794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:21.456796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:21.457110 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:21.457215 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:21.458206 (+ 991us) server_negotiation.cc:299] Negotiation successful
0924 07:08:21.458410 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:08:21.461171 17027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:21.437579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:38767 (local address 127.3.217.2:60517)
0924 07:08:21.439785 (+ 2206us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:21.439837 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:21.441228 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:21.442366 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:21.442378 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:21.442817 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:21.443675 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:21.443698 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:21.444901 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:21.444904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:21.446338 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:21.446353 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:21.447193 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:21.447197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:21.447453 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:21.450887 (+ 3434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:21.450914 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:21.452808 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:21.455194 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:21.455206 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:21.455234 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:21.455865 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:21.456378 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:21.456382 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:21.456384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:21.456655 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:21.457302 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:21.457310 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:21.457829 (+ 519us) client_negotiation.cc:769] Sending connection context
0924 07:08:21.458943 (+ 1114us) client_negotiation.cc:240] Negotiation successful
0924 07:08:21.459704 (+ 761us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1996,"thread_start_us":132,"threads_started":1}
I20260924 07:08:21.462946 17025 heartbeater.cc:347] Connected to a master server at 127.3.217.62:38767
I20260924 07:08:21.463929 17025 heartbeater.cc:464] Registering TS with master...
I20260924 07:08:21.464901 17025 heartbeater.cc:511] Master 127.3.217.62:38767 requested a full tablet report, sending...
W20260924 07:08:21.480002 17025 heartbeater.cc:644] Failed to heartbeat to 127.3.217.62:38767 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 07:08:21.480206 17025 heartbeater.cc:464] Registering TS with master...
I20260924 07:08:21.480489 17025 heartbeater.cc:511] Master 127.3.217.62:38767 requested a full tablet report, sending...
I20260924 07:08:21.482364 17025 heartbeater.cc:464] Registering TS with master...
I20260924 07:08:21.482702 17025 heartbeater.cc:511] Master 127.3.217.62:38767 requested a full tablet report, sending...
W20260924 07:08:21.483955 17025 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:08:21.642724 17028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:21.643069 17028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:21.643134 17028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:21.649757 17028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:21.649856 17028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:21.649969 17028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:08:21.659268 17028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:44009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.217.20:39933/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:39585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=37347
--tserver_master_addrs=127.3.217.62:38767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:21.660687 17028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:21.662045 17028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:21.676349 17035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:21.676597 17034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:21.676734 17037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:21.676800 17028 server_base.cc:1034] running on GCE node
I20260924 07:08:21.677239 17028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:21.677960 17028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:21.681396 17028 hybrid_clock.cc:648] HybridClock initialized: now 1790233701681325 us; error 51 us; skew 500 ppm
Sep 24 07:08:21 dist-test-slave-2p7j krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233701, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:21.688827 17028 init.cc:377] Logged in from keytab as kudu/127.3.217.3@KRBTEST.COM (short username kudu)
I20260924 07:08:21.690083 17028 webserver.cc:460] Webserver started at http://127.3.217.3:37347/ using document root <none> and password file <none>
I20260924 07:08:21.690814 17028 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:21.690929 17028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 07:08:21 dist-test-slave-2p7j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233701, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for HTTP/127.3.217.20@KRBTEST.COM
I20260924 07:08:21.716521 17028 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.000s
I20260924 07:08:21.719316 17045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:21.721055 17028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:08:21.721190 17028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "902c1f58a0134eb1b9c7f6fa938c2548"
format_stamp: "Formatted at 2026-09-24 07:08:20 on dist-test-slave-2p7j"
server_key: "54e8c2b9c758b0c8c5e890b890c16197"
server_key_iv: "cf23a25fb55f8f60de8acf4822e3d985"
server_key_version: "kuduclusterkey@0"
I20260924 07:08:21.724049 17028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:21.752671 17028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:21.761250 17028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:21.761567 17028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:21.762344 17028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:08:21.763280 17028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:08:21.763375 17028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:21.763441 17028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:08:21.763476 17028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:21.777256 17028 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:39585
I20260924 07:08:21.777705 17158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:39585 every 8 connection(s)
I20260924 07:08:21.778995 17028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:08:21.787719 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 17028
Sep 24 07:08:21 dist-test-slave-2p7j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233701, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:08:21.800918 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:21.781776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:40555 (local address 127.3.217.62:38767)
0924 07:08:21.781929 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:21.781945 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:21.783751 (+ 1806us) server_negotiation.cc:407] Connection header received
0924 07:08:21.785007 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:21.785011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:21.785068 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:21.785163 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:21.786875 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:21.787747 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:21.789589 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:21.789892 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:21.789990 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:21.795678 (+ 5688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:21.795682 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:21.795685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:21.795714 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:21.797549 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:21.798235 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:21.798238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:21.798240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:21.798413 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:21.798879 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:21.798883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:21.798884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:21.799334 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:21.799629 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:21.800361 (+ 732us) server_negotiation.cc:299] Negotiation successful
0924 07:08:21.800576 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50,"spinlock_wait_cycles":9984}
I20260924 07:08:21.801789 17161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:21.782252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:38767 (local address 127.3.217.3:40555)
0924 07:08:21.783567 (+ 1315us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:21.783627 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:21.784766 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:21.785382 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:21.785392 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:21.785760 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:21.786691 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:21.786714 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:21.787897 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:21.787901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:21.789472 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:21.789482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:21.790149 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:21.790154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:21.790345 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:21.791632 (+ 1287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:21.791664 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:21.795448 (+ 3784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:21.797676 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:21.797684 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:21.797705 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:21.798119 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:21.798543 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:21.798546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:21.798548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:21.798755 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:21.799566 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:21.799574 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:21.799964 (+ 390us) client_negotiation.cc:769] Sending connection context
0924 07:08:21.800221 (+ 257us) client_negotiation.cc:240] Negotiation successful
0924 07:08:21.800608 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"thread_start_us":213,"threads_started":1}
I20260924 07:08:21.802692 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:21.788437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.3.217.62:38767)
0924 07:08:21.788694 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:21.788698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:21.789912 (+ 1214us) server_negotiation.cc:407] Connection header received
0924 07:08:21.789944 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:21.789947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:21.789992 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:21.790057 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:21.790937 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:21.792655 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:21.796085 (+ 3430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:21.796559 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:21.796797 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:21.798309 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:21.798312 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:21.798314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:21.798335 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:21.799795 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:21.800257 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:21.800260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:21.800261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:21.800345 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:21.800645 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:21.800648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:21.800650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:21.800943 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:21.801441 (+ 498us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:21.802387 (+ 946us) server_negotiation.cc:299] Negotiation successful
0924 07:08:21.802539 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":81,"threads_started":1}
I20260924 07:08:21.803237 17159 heartbeater.cc:347] Connected to a master server at 127.3.217.62:38767
I20260924 07:08:21.803753 17159 heartbeater.cc:464] Registering TS with master...
I20260924 07:08:21.804473 17159 heartbeater.cc:511] Master 127.3.217.62:38767 requested a full tablet report, sending...
W20260924 07:08:21.807647 17159 heartbeater.cc:644] Failed to heartbeat to 127.3.217.62:38767 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 07:08:21.807780 17159 heartbeater.cc:464] Registering TS with master...
I20260924 07:08:21.808255 17159 heartbeater.cc:511] Master 127.3.217.62:38767 requested a full tablet report, sending...
I20260924 07:08:21.810825 17159 heartbeater.cc:464] Registering TS with master...
I20260924 07:08:21.811051 17159 heartbeater.cc:511] Master 127.3.217.62:38767 requested a full tablet report, sending...
W20260924 07:08:21.812597 17159 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 07:08:22.150995 16891 heartbeater.cc:464] Registering TS with master...
I20260924 07:08:22.151283 16891 heartbeater.cc:511] Master 127.3.217.62:38767 requested a full tablet report, sending...
I20260924 07:08:22.153126 16693 ts_manager.cc:194] Registered new tserver with Master: 753452748bfb45188ab0f1b65c9be6de (127.3.217.1:36317)
I20260924 07:08:22.154542 16693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.1@KRBTEST.COM'} at 127.3.217.1:38125
Sep 24 07:08:22 dist-test-slave-2p7j krb5kdc[15696](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790233702, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:22.484287 17025 heartbeater.cc:464] Registering TS with master...
I20260924 07:08:22.484643 17025 heartbeater.cc:511] Master 127.3.217.62:38767 requested a full tablet report, sending...
I20260924 07:08:22.485404 16693 ts_manager.cc:194] Registered new tserver with Master: 138b349fb78248f29c1fe0aa0ea86c7b (127.3.217.2:43945)
I20260924 07:08:22.486178 16693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.2@KRBTEST.COM'} at 127.3.217.2:60517
Sep 24 07:08:22 dist-test-slave-2p7j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233702, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for HTTP/127.3.217.20@KRBTEST.COM
I20260924 07:08:22.812913 17159 heartbeater.cc:464] Registering TS with master...
I20260924 07:08:22.813211 17159 heartbeater.cc:511] Master 127.3.217.62:38767 requested a full tablet report, sending...
I20260924 07:08:22.813911 16693 ts_manager.cc:194] Registered new tserver with Master: 902c1f58a0134eb1b9c7f6fa938c2548 (127.3.217.3:39585)
I20260924 07:08:22.814761 16693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.3@KRBTEST.COM'} at 127.3.217.3:40555
I20260924 07:08:22.824479 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:08:22.837405 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:22.826077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.3.217.62:38767)
0924 07:08:22.826320 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:22.826324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:22.827877 (+ 1553us) server_negotiation.cc:407] Connection header received
0924 07:08:22.827983 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:22.827986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:22.828040 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:22.828146 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:22.828921 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:22.829723 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:22.830967 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:22.831635 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:22.831814 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:22.833662 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:22.833666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:22.833669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:22.833700 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:22.835462 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:22.835964 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:22.835968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:22.835970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:22.836052 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:22.836327 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:22.836330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:22.836332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:22.836543 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:22.836638 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:22.837091 (+ 453us) server_negotiation.cc:299] Negotiation successful
0924 07:08:22.837228 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":89,"threads_started":1}
I20260924 07:08:22.842865 16693 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:08:22.847608 16693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:08:22.868964 17190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:22.861359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:43945 (local address 127.0.0.1:37274)
0924 07:08:22.862585 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:22.862599 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:22.862682 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:22.864147 (+ 1465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:22.864150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:22.864168 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:08:22.864260 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:22.864266 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:22.865664 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:22.865668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:22.866775 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:22.866786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:22.867820 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:22.867824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:22.868529 (+ 705us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:22.868578 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 07:08:22.868684 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 07:08:22.868804 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1172,"mutex_wait_us":572,"thread_start_us":84,"threads_started":1}
I20260924 07:08:22.869634 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:22.859797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:36317 (local address 127.0.0.1:41684)
0924 07:08:22.860225 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:22.860273 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:22.860400 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:22.862363 (+ 1963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:22.862369 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:22.862448 (+ 79us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:08:22.862605 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:22.862613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:22.864918 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:22.864921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:22.866294 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:22.866304 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:22.867259 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:22.867263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:22.869342 (+ 2079us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:22.869366 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 07:08:22.869439 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 07:08:22.869510 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":815,"thread_start_us":120,"threads_started":1}
I20260924 07:08:22.870069 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:22.860624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41684 (local address 127.3.217.1:36317)
0924 07:08:22.861083 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:22.861090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:22.861114 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:08:22.861187 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:22.861192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:22.862197 (+ 1005us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:22.862757 (+ 560us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:08:22.862891 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:22.864801 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:22.866433 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:22.867134 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:22.868553 (+ 1419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:22.868778 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:22.869523 (+ 745us) server_negotiation.cc:299] Negotiation successful
0924 07:08:22.869872 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":94,"threads_started":1}
I20260924 07:08:22.870151 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:22.861516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37274 (local address 127.3.217.2:43945)
0924 07:08:22.863711 (+ 2195us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:22.863717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:22.863740 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:08:22.863809 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:22.863814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:22.863985 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:22.864388 (+ 403us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:08:22.864523 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:22.865531 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:22.866918 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:22.867694 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:22.867869 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:22.868013 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:22.869894 (+ 1881us) server_negotiation.cc:299] Negotiation successful
0924 07:08:22.870002 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2098,"thread_start_us":106,"threads_started":1}
I20260924 07:08:22.873005 16959 tablet_service.cc:1467] Processing CreateTablet for tablet 41674fd5be4440c69471ccfead17b92a (DEFAULT_TABLE table=test-table [id=d26cb25477ce42f592f9d5577be230d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:08:22.873296 16825 tablet_service.cc:1467] Processing CreateTablet for tablet 41674fd5be4440c69471ccfead17b92a (DEFAULT_TABLE table=test-table [id=d26cb25477ce42f592f9d5577be230d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:08:22.873978 16959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41674fd5be4440c69471ccfead17b92a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:08:22.874158 16825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41674fd5be4440c69471ccfead17b92a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:08:22.888804 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:22.861961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:39585 (local address 127.0.0.1:35656)
0924 07:08:22.862829 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:22.862839 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:22.862896 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:22.870947 (+ 8051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:22.870952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:22.870983 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:08:22.871117 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:22.871124 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:22.878652 (+ 7528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:22.878656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:22.879898 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:22.879910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:22.888229 (+ 8319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:22.888235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:22.888432 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:22.888463 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 07:08:22.888552 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 07:08:22.888640 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"thread_start_us":73,"threads_started":1}
I20260924 07:08:22.888846 17193 tablet_bootstrap.cc:492] T 41674fd5be4440c69471ccfead17b92a P 753452748bfb45188ab0f1b65c9be6de: Bootstrap starting.
I20260924 07:08:22.889394 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:22.862405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.3.217.3:39585)
0924 07:08:22.869752 (+ 7347us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:22.869760 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:22.869792 (+ 32us) server_negotiation.cc:407] Connection header received
0924 07:08:22.869928 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:22.869934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:22.870236 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:22.871415 (+ 1179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:08:22.877306 (+ 5891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:22.878528 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:22.886191 (+ 7663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:22.887972 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:22.888125 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:22.888312 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:22.889126 (+ 814us) server_negotiation.cc:299] Negotiation successful
0924 07:08:22.889254 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7173,"thread_start_us":82,"threads_started":1}
I20260924 07:08:22.892438 17193 tablet_bootstrap.cc:654] T 41674fd5be4440c69471ccfead17b92a P 753452748bfb45188ab0f1b65c9be6de: Neither blocks nor log segments found. Creating new log.
I20260924 07:08:22.893561 17193 log.cc:824] T 41674fd5be4440c69471ccfead17b92a P 753452748bfb45188ab0f1b65c9be6de: Log is configured to *not* fsync() on all Append() calls
I20260924 07:08:22.893597 17093 tablet_service.cc:1467] Processing CreateTablet for tablet 41674fd5be4440c69471ccfead17b92a (DEFAULT_TABLE table=test-table [id=d26cb25477ce42f592f9d5577be230d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:08:22.894534 17192 tablet_bootstrap.cc:492] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b: Bootstrap starting.
I20260924 07:08:22.895058 17093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41674fd5be4440c69471ccfead17b92a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:08:22.897859 17192 tablet_bootstrap.cc:654] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b: Neither blocks nor log segments found. Creating new log.
I20260924 07:08:22.898993 17192 log.cc:824] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b: Log is configured to *not* fsync() on all Append() calls
I20260924 07:08:22.903306 17193 tablet_bootstrap.cc:492] T 41674fd5be4440c69471ccfead17b92a P 753452748bfb45188ab0f1b65c9be6de: No bootstrap required, opened a new log
I20260924 07:08:22.903669 17193 ts_tablet_manager.cc:1397] T 41674fd5be4440c69471ccfead17b92a P 753452748bfb45188ab0f1b65c9be6de: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20260924 07:08:22.907793 17193 raft_consensus.cc:348] T 41674fd5be4440c69471ccfead17b92a P 753452748bfb45188ab0f1b65c9be6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753452748bfb45188ab0f1b65c9be6de" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 36317 } } peers { permanent_uuid: "138b349fb78248f29c1fe0aa0ea86c7b" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43945 } } peers { permanent_uuid: "902c1f58a0134eb1b9c7f6fa938c2548" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 39585 } }
I20260924 07:08:22.908095 17193 raft_consensus.cc:374] T 41674fd5be4440c69471ccfead17b92a P 753452748bfb45188ab0f1b65c9be6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:08:22.908155 17193 raft_consensus.cc:729] T 41674fd5be4440c69471ccfead17b92a P 753452748bfb45188ab0f1b65c9be6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 753452748bfb45188ab0f1b65c9be6de, State: Initialized, Role: FOLLOWER
I20260924 07:08:22.910413 17192 tablet_bootstrap.cc:492] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b: No bootstrap required, opened a new log
I20260924 07:08:22.910638 17192 ts_tablet_manager.cc:1397] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.004s
I20260924 07:08:22.915915 17193 consensus_queue.cc:260] T 41674fd5be4440c69471ccfead17b92a P 753452748bfb45188ab0f1b65c9be6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753452748bfb45188ab0f1b65c9be6de" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 36317 } } peers { permanent_uuid: "138b349fb78248f29c1fe0aa0ea86c7b" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43945 } } peers { permanent_uuid: "902c1f58a0134eb1b9c7f6fa938c2548" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 39585 } }
I20260924 07:08:22.915683 17192 raft_consensus.cc:348] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753452748bfb45188ab0f1b65c9be6de" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 36317 } } peers { permanent_uuid: "138b349fb78248f29c1fe0aa0ea86c7b" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43945 } } peers { permanent_uuid: "902c1f58a0134eb1b9c7f6fa938c2548" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 39585 } }
I20260924 07:08:22.916193 17192 raft_consensus.cc:374] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:08:22.916270 17192 raft_consensus.cc:729] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138b349fb78248f29c1fe0aa0ea86c7b, State: Initialized, Role: FOLLOWER
I20260924 07:08:22.916780 17192 consensus_queue.cc:260] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753452748bfb45188ab0f1b65c9be6de" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 36317 } } peers { permanent_uuid: "138b349fb78248f29c1fe0aa0ea86c7b" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43945 } } peers { permanent_uuid: "902c1f58a0134eb1b9c7f6fa938c2548" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 39585 } }
I20260924 07:08:22.917426 17193 ts_tablet_manager.cc:1428] T 41674fd5be4440c69471ccfead17b92a P 753452748bfb45188ab0f1b65c9be6de: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260924 07:08:22.917774 17192 ts_tablet_manager.cc:1428] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b: Time spent starting tablet: real 0.007s user 0.008s sys 0.001s
I20260924 07:08:22.917888 16891 heartbeater.cc:503] Master 127.3.217.62:38767 was elected leader, sending a full tablet report...
I20260924 07:08:22.918109 17025 heartbeater.cc:503] Master 127.3.217.62:38767 was elected leader, sending a full tablet report...
I20260924 07:08:22.919265 17196 tablet_bootstrap.cc:492] T 41674fd5be4440c69471ccfead17b92a P 902c1f58a0134eb1b9c7f6fa938c2548: Bootstrap starting.
I20260924 07:08:22.921824 17196 tablet_bootstrap.cc:654] T 41674fd5be4440c69471ccfead17b92a P 902c1f58a0134eb1b9c7f6fa938c2548: Neither blocks nor log segments found. Creating new log.
I20260924 07:08:22.924465 17196 log.cc:824] T 41674fd5be4440c69471ccfead17b92a P 902c1f58a0134eb1b9c7f6fa938c2548: Log is configured to *not* fsync() on all Append() calls
I20260924 07:08:22.927418 17196 tablet_bootstrap.cc:492] T 41674fd5be4440c69471ccfead17b92a P 902c1f58a0134eb1b9c7f6fa938c2548: No bootstrap required, opened a new log
I20260924 07:08:22.927675 17196 ts_tablet_manager.cc:1397] T 41674fd5be4440c69471ccfead17b92a P 902c1f58a0134eb1b9c7f6fa938c2548: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 07:08:22.931931 17196 raft_consensus.cc:348] T 41674fd5be4440c69471ccfead17b92a P 902c1f58a0134eb1b9c7f6fa938c2548 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753452748bfb45188ab0f1b65c9be6de" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 36317 } } peers { permanent_uuid: "138b349fb78248f29c1fe0aa0ea86c7b" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43945 } } peers { permanent_uuid: "902c1f58a0134eb1b9c7f6fa938c2548" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 39585 } }
I20260924 07:08:22.932206 17196 raft_consensus.cc:374] T 41674fd5be4440c69471ccfead17b92a P 902c1f58a0134eb1b9c7f6fa938c2548 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:08:22.932269 17196 raft_consensus.cc:729] T 41674fd5be4440c69471ccfead17b92a P 902c1f58a0134eb1b9c7f6fa938c2548 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 902c1f58a0134eb1b9c7f6fa938c2548, State: Initialized, Role: FOLLOWER
I20260924 07:08:22.932760 17196 consensus_queue.cc:260] T 41674fd5be4440c69471ccfead17b92a P 902c1f58a0134eb1b9c7f6fa938c2548 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753452748bfb45188ab0f1b65c9be6de" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 36317 } } peers { permanent_uuid: "138b349fb78248f29c1fe0aa0ea86c7b" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43945 } } peers { permanent_uuid: "902c1f58a0134eb1b9c7f6fa938c2548" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 39585 } }
I20260924 07:08:22.933715 17159 heartbeater.cc:503] Master 127.3.217.62:38767 was elected leader, sending a full tablet report...
I20260924 07:08:22.933845 17196 ts_tablet_manager.cc:1428] T 41674fd5be4440c69471ccfead17b92a P 902c1f58a0134eb1b9c7f6fa938c2548: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260924 07:08:22.939848 17026 tablet.cc:2369] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:08:23.031754 17160 tablet.cc:2369] T 41674fd5be4440c69471ccfead17b92a P 902c1f58a0134eb1b9c7f6fa938c2548: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 07:08:23 dist-test-slave-2p7j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233702, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for HTTP/127.3.217.20@KRBTEST.COM
I20260924 07:08:23.062165 17198 raft_consensus.cc:482] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:08:23.062408 17198 raft_consensus.cc:504] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753452748bfb45188ab0f1b65c9be6de" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 36317 } } peers { permanent_uuid: "138b349fb78248f29c1fe0aa0ea86c7b" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43945 } } peers { permanent_uuid: "902c1f58a0134eb1b9c7f6fa938c2548" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 39585 } }
I20260924 07:08:23.063772 17198 leader_election.cc:288] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 753452748bfb45188ab0f1b65c9be6de (127.3.217.1:36317), 902c1f58a0134eb1b9c7f6fa938c2548 (127.3.217.3:39585)
I20260924 07:08:23.072132 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:23.064167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:51187 (local address 127.3.217.1:36317)
0924 07:08:23.064301 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:23.064306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:23.067643 (+ 3337us) server_negotiation.cc:407] Connection header received
0924 07:08:23.067818 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:23.067821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:23.067872 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:23.067915 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:08:23.068285 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:23.069046 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:23.070789 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:23.071386 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:23.071463 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:23.071567 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:23.071898 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 07:08:23.071978 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:08:23.072618 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:23.064092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:36317 (local address 127.3.217.2:51187)
0924 07:08:23.067564 (+ 3472us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:23.067577 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:23.067707 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:23.068042 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:23.068045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:23.068061 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:08:23.068174 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:23.068179 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:23.069183 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:23.069186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:23.070644 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:23.070655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:23.071659 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:23.071663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:23.071768 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:23.071791 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:08:23.072363 (+ 572us) client_negotiation.cc:240] Negotiation successful
0924 07:08:23.072439 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3364,"mutex_wait_us":83,"thread_start_us":148,"threads_started":1}
I20260924 07:08:23.073268 16845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41674fd5be4440c69471ccfead17b92a" candidate_uuid: "138b349fb78248f29c1fe0aa0ea86c7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "753452748bfb45188ab0f1b65c9be6de" is_pre_election: true
I20260924 07:08:23.073617 16845 raft_consensus.cc:2454] T 41674fd5be4440c69471ccfead17b92a P 753452748bfb45188ab0f1b65c9be6de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 138b349fb78248f29c1fe0aa0ea86c7b in term 0.
I20260924 07:08:23.074301 16915 leader_election.cc:302] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 138b349fb78248f29c1fe0aa0ea86c7b, 753452748bfb45188ab0f1b65c9be6de; no voters:
I20260924 07:08:23.074630 17198 raft_consensus.cc:2781] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:08:23.074712 17198 raft_consensus.cc:482] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:08:23.074757 17198 raft_consensus.cc:3037] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:08:23.075413 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:23.064527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:32907 (local address 127.3.217.3:39585)
0924 07:08:23.064656 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:23.064660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:23.067278 (+ 2618us) server_negotiation.cc:407] Connection header received
0924 07:08:23.067448 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:23.067451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:23.067541 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:23.067616 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:08:23.068029 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:23.068783 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:23.071255 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:23.072058 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:23.072141 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:23.072199 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:23.075193 (+ 2994us) server_negotiation.cc:299] Negotiation successful
0924 07:08:23.075289 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:08:23.075784 17202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:23.064049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:39585 (local address 127.3.217.2:32907)
0924 07:08:23.067184 (+ 3135us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:23.067199 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:23.067325 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:23.067737 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:23.067741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:23.067759 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:08:23.067900 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:23.067907 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:23.068914 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:23.068918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:23.070487 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:23.070515 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:23.074940 (+ 4425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:23.074944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:23.075055 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:23.075078 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:08:23.075572 (+ 494us) client_negotiation.cc:240] Negotiation successful
0924 07:08:23.075657 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3056,"thread_start_us":162,"threads_started":1}
I20260924 07:08:23.076144 17198 raft_consensus.cc:504] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753452748bfb45188ab0f1b65c9be6de" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 36317 } } peers { permanent_uuid: "138b349fb78248f29c1fe0aa0ea86c7b" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43945 } } peers { permanent_uuid: "902c1f58a0134eb1b9c7f6fa938c2548" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 39585 } }
I20260924 07:08:23.076524 17198 leader_election.cc:288] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b [CANDIDATE]: Term 1 election: Requested vote from peers 753452748bfb45188ab0f1b65c9be6de (127.3.217.1:36317), 902c1f58a0134eb1b9c7f6fa938c2548 (127.3.217.3:39585)
I20260924 07:08:23.077353 17113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41674fd5be4440c69471ccfead17b92a" candidate_uuid: "138b349fb78248f29c1fe0aa0ea86c7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "902c1f58a0134eb1b9c7f6fa938c2548" is_pre_election: true
I20260924 07:08:23.077540 17112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41674fd5be4440c69471ccfead17b92a" candidate_uuid: "138b349fb78248f29c1fe0aa0ea86c7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "902c1f58a0134eb1b9c7f6fa938c2548"
I20260924 07:08:23.077653 17113 raft_consensus.cc:2454] T 41674fd5be4440c69471ccfead17b92a P 902c1f58a0134eb1b9c7f6fa938c2548 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 138b349fb78248f29c1fe0aa0ea86c7b in term 0.
I20260924 07:08:23.078706 16845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41674fd5be4440c69471ccfead17b92a" candidate_uuid: "138b349fb78248f29c1fe0aa0ea86c7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "753452748bfb45188ab0f1b65c9be6de"
I20260924 07:08:23.078840 16845 raft_consensus.cc:3037] T 41674fd5be4440c69471ccfead17b92a P 753452748bfb45188ab0f1b65c9be6de [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:08:23.080199 16845 raft_consensus.cc:2454] T 41674fd5be4440c69471ccfead17b92a P 753452748bfb45188ab0f1b65c9be6de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 138b349fb78248f29c1fe0aa0ea86c7b in term 1.
I20260924 07:08:23.080551 16915 leader_election.cc:302] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 138b349fb78248f29c1fe0aa0ea86c7b, 753452748bfb45188ab0f1b65c9be6de; no voters: 902c1f58a0134eb1b9c7f6fa938c2548
I20260924 07:08:23.080793 17198 raft_consensus.cc:2781] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:08:23.081000 17198 raft_consensus.cc:686] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b [term 1 LEADER]: Becoming Leader. State: Replica: 138b349fb78248f29c1fe0aa0ea86c7b, State: Running, Role: LEADER
I20260924 07:08:23.081352 17198 consensus_queue.cc:237] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753452748bfb45188ab0f1b65c9be6de" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 36317 } } peers { permanent_uuid: "138b349fb78248f29c1fe0aa0ea86c7b" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43945 } } peers { permanent_uuid: "902c1f58a0134eb1b9c7f6fa938c2548" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 39585 } }
I20260924 07:08:23.086449 16691 catalog_manager.cc:5495] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 138b349fb78248f29c1fe0aa0ea86c7b (127.3.217.2). New cstate: current_term: 1 leader_uuid: "138b349fb78248f29c1fe0aa0ea86c7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753452748bfb45188ab0f1b65c9be6de" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 36317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "138b349fb78248f29c1fe0aa0ea86c7b" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 43945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "902c1f58a0134eb1b9c7f6fa938c2548" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 39585 } health_report { overall_health: UNKNOWN } } }
W20260924 07:08:23.108492 16892 tablet.cc:2369] T 41674fd5be4440c69471ccfead17b92a P 753452748bfb45188ab0f1b65c9be6de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:08:23.109683 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:23.104908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37286 (local address 127.3.217.2:43945)
0924 07:08:23.105025 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:23.105030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:23.105042 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:08:23.105115 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:23.105117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:23.105168 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:23.105254 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:08:23.105508 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:23.106285 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:23.107549 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:23.107878 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:23.107982 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:23.108416 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:08:23.108942 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:08:23.109071 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:23.109467 (+ 396us) server_negotiation.cc:299] Negotiation successful
0924 07:08:23.109530 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:08:23.117679 17112 raft_consensus.cc:3037] T 41674fd5be4440c69471ccfead17b92a P 902c1f58a0134eb1b9c7f6fa938c2548 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:08:23.119232 17112 raft_consensus.cc:1261] T 41674fd5be4440c69471ccfead17b92a P 902c1f58a0134eb1b9c7f6fa938c2548 [term 1 FOLLOWER]: Refusing update from remote peer 138b349fb78248f29c1fe0aa0ea86c7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:08:23.119944 17204 consensus_queue.cc:1035] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "902c1f58a0134eb1b9c7f6fa938c2548" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 39585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:08:23.121618 16845 raft_consensus.cc:1261] T 41674fd5be4440c69471ccfead17b92a P 753452748bfb45188ab0f1b65c9be6de [term 1 FOLLOWER]: Refusing update from remote peer 138b349fb78248f29c1fe0aa0ea86c7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:08:23.122233 17204 consensus_queue.cc:1035] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "753452748bfb45188ab0f1b65c9be6de" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:08:23.137924 17208 mvcc.cc:205] Tried to move back new op lower bound from 7332797247957880832 to 7332797247824846848. Current Snapshot: MvccSnapshot[applied={T|T < 7332797247957880832}]
I20260924 07:08:23.141227 17209 mvcc.cc:205] Tried to move back new op lower bound from 7332797247957880832 to 7332797247824846848. Current Snapshot: MvccSnapshot[applied={T|T < 7332797247957880832}]
I20260924 07:08:23.145707 17211 mvcc.cc:205] Tried to move back new op lower bound from 7332797247957880832 to 7332797247824846848. Current Snapshot: MvccSnapshot[applied={T|T < 7332797247957880832}]
I20260924 07:08:23.164120 16691 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:08:23.164297 16691 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:08:23.166926 16691 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3bfbf3b0cdc4e52b8463453d0406773: Sending DeleteTablet for 3 replicas of tablet 41674fd5be4440c69471ccfead17b92a
I20260924 07:08:23.169486 16825 tablet_service.cc:1514] Processing DeleteTablet for tablet 41674fd5be4440c69471ccfead17b92a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:08:23 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:41684
I20260924 07:08:23.169638 17093 tablet_service.cc:1514] Processing DeleteTablet for tablet 41674fd5be4440c69471ccfead17b92a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:08:23 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:35656
I20260924 07:08:23.170454 16959 tablet_service.cc:1514] Processing DeleteTablet for tablet 41674fd5be4440c69471ccfead17b92a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:08:23 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:37274
I20260924 07:08:23.171991 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 16731
W20260924 07:08:23.194099 16680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.217.1:36317 (error 108)
I20260924 07:08:23.194645 17218 tablet_replica.cc:330] T 41674fd5be4440c69471ccfead17b92a P 902c1f58a0134eb1b9c7f6fa938c2548: stopping tablet replica
I20260924 07:08:23.195019 17219 tablet_replica.cc:330] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b: stopping tablet replica
I20260924 07:08:23.195612 17219 raft_consensus.cc:2229] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:08:23.196007 17218 raft_consensus.cc:2229] T 41674fd5be4440c69471ccfead17b92a P 902c1f58a0134eb1b9c7f6fa938c2548 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:08:23.196430 17218 raft_consensus.cc:2258] T 41674fd5be4440c69471ccfead17b92a P 902c1f58a0134eb1b9c7f6fa938c2548 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:08:23.196866 17219 raft_consensus.cc:2258] T 41674fd5be4440c69471ccfead17b92a P 138b349fb78248f29c1fe0aa0ea86c7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:08:23.198589 17218 ts_tablet_manager.cc:1905] T 41674fd5be4440c69471ccfead17b92a P 902c1f58a0134eb1b9c7f6fa938c2548: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:08:23.204537 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 16895
I20260924 07:08:23.208021 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:23.207628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:36317 (local address 127.0.0.1:41694)
0924 07:08:23.207837 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:23.207911 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.1:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260924 07:08:23.208276 16680 catalog_manager.cc:4584] TS 753452748bfb45188ab0f1b65c9be6de (127.3.217.1:36317): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 41674fd5be4440c69471ccfead17b92a: Network error: Client connection negotiation failed: client connection to 127.3.217.1:36317: connect: Connection refused (error 111)
W20260924 07:08:23.215204 17188 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:08:23.214585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:43945 (local address 127.0.0.1:37294)
0924 07:08:23.214745 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:23.214759 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:23.214843 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:23.215082 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.2:43945: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 07:08:23.216070 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 17028
I20260924 07:08:23.229415 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 16660
I20260924 07:08:23.229597 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:23.229284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:39585 (local address 127.0.0.1:35664)
0924 07:08:23.229455 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:23.229510 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.3:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260924 07:08:23.238643 3940 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 07:08:23.574502 3940 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 07:08:23.574779 3940 mini_ranger.cc:66] Stopping Ranger...
I20260924 07:08:23.921893 3940 mini_ranger.cc:68] Stopped Ranger
2026-09-24 07:08:23.922 UTC [15726] LOG: received smart shutdown request
2026-09-24 07:08:23.927 UTC [15726] LOG: background worker "logical replication launcher" (PID 15735) exited with exit code 1
2026-09-24 07:08:23.927 UTC [15730] LOG: shutting down
2026-09-24 07:08:24.148 UTC [15726] LOG: database system is shut down
2026-09-24T07:08:24Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50807 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:08:24 dist-test-slave-2p7j krb5kdc[17229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:08:24 dist-test-slave-2p7j krb5kdc[17229](info): set up 2 sockets
Sep 24 07:08:24 dist-test-slave-2p7j krb5kdc[17229](info): commencing operation
krb5kdc: starting...
W20260924 07:08:25.269539 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:08:25 dist-test-slave-2p7j krb5kdc[17229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:08:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:08:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:08:25.416069 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:32975
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:32975,127.3.217.61:45143,127.3.217.60:35929
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:08:25.525484 17242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:25.525764 17242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:25.525820 17242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:25.529843 17242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:08:25.529898 17242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:25.529923 17242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:25.529945 17242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:08:25.529974 17242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:08:25.535388 17242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:32975,127.3.217.61:45143,127.3.217.60:35929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:32975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:25.536419 17242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:25.537387 17242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:25.545672 17250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:25.545656 17247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:25.545656 17248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:25.545727 17242 server_base.cc:1034] running on GCE node
I20260924 07:08:25.546613 17242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:25.547667 17242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:25.548877 17242 hybrid_clock.cc:648] HybridClock initialized: now 1790233705548858 us; error 54 us; skew 500 ppm
Sep 24 07:08:25 dist-test-slave-2p7j krb5kdc[17229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233705, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:25.557958 17242 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:08:25.558894 17242 webserver.cc:460] Webserver started at http://127.3.217.62:46691/ using document root <none> and password file <none>
I20260924 07:08:25.559422 17242 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:25.559485 17242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:08:25.559712 17242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:08:25.561748 17242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "90cda92a4f4e4e29812187820864d26a"
format_stamp: "Formatted at 2026-09-24 07:08:25 on dist-test-slave-2p7j"
server_key: "98b93bd84b50acd5c6d66cc10c8b4891"
server_key_iv: "c2a34fc7a49c42ac3c8918501dc71f5a"
server_key_version: "encryptionkey@0"
I20260924 07:08:25.562315 17242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "90cda92a4f4e4e29812187820864d26a"
format_stamp: "Formatted at 2026-09-24 07:08:25 on dist-test-slave-2p7j"
server_key: "98b93bd84b50acd5c6d66cc10c8b4891"
server_key_iv: "c2a34fc7a49c42ac3c8918501dc71f5a"
server_key_version: "encryptionkey@0"
I20260924 07:08:25.566028 17242 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 07:08:25.569823 17257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:25.571331 17242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:08:25.571457 17242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "90cda92a4f4e4e29812187820864d26a"
format_stamp: "Formatted at 2026-09-24 07:08:25 on dist-test-slave-2p7j"
server_key: "98b93bd84b50acd5c6d66cc10c8b4891"
server_key_iv: "c2a34fc7a49c42ac3c8918501dc71f5a"
server_key_version: "encryptionkey@0"
I20260924 07:08:25.571640 17242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:25.583693 17242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:25.586642 17242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:25.586843 17242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:25.594573 17242 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:32975
I20260924 07:08:25.594609 17309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:32975 every 8 connection(s)
I20260924 07:08:25.595494 17242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:08:25.598193 17310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:08:25.601825 17310 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.62" port: 32975 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:08:25.602741 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 17242
I20260924 07:08:25.602854 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:08:25.603158 3940 external_mini_cluster.cc:1428] Setting key b29311f2617a86ffecfc46eb26a162bb
WARNING: no policy specified for kudu/127.3.217.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.61@KRBTEST.COM" created.
Sep 24 07:08:25 dist-test-slave-2p7j krb5kdc[17229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233705, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:08:25.619956 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.604262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:32975 (local address 127.0.0.1:34720)
0924 07:08:25.604750 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:25.604799 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:25.605666 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:25.606455 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:25.606459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:25.606781 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:25.607370 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:25.607397 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:25.609287 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:25.609291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:25.610269 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:25.610282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:25.611237 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:25.611241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:25.611337 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:25.611924 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:25.611940 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:25.614784 (+ 2844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:25.616825 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:25.616835 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:25.616845 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:25.617126 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:25.617534 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:25.617537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:25.617539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:25.617686 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:25.618342 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:25.618349 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:25.618522 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 07:08:25.618718 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 07:08:25.619129 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":93,"thread_start_us":144,"threads_started":1}
I20260924 07:08:25.619959 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.605102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34720 (local address 127.3.217.62:32975)
0924 07:08:25.605402 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:25.605412 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:25.605455 (+ 43us) server_negotiation.cc:407] Connection header received
0924 07:08:25.605974 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:25.605980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:25.606220 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:25.606408 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:25.607654 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:25.609018 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:25.610434 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:25.611079 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:25.611216 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:25.614948 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:25.614954 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:25.614966 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:25.614987 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:25.616703 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:25.617310 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:25.617316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:25.617317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:25.617428 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:25.617801 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:25.617804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:25.617805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:25.618213 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:25.618334 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:25.618770 (+ 436us) server_negotiation.cc:299] Negotiation successful
0924 07:08:25.619014 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":89,"threads_started":1}
I20260924 07:08:25.622715 17310 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:08:25.623732 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.623144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:45143 (local address 127.0.0.1:56036)
0924 07:08:25.623365 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:25.623580 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.61:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260924 07:08:25.623981 17260 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.217.61:45143: connect: Connection refused (error 111)
I20260924 07:08:25.624783 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.624567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:45143 (local address 127.0.0.1:56038)
0924 07:08:25.624689 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:25.624736 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.61:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 07:08:25.624912 17310 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.61:45143: Network error: Client connection negotiation failed: client connection to 127.3.217.61:45143: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.61.keytab.
Entry for principal kudu/127.3.217.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.61.keytab.
I20260924 07:08:25.631343 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.61:45143
--webserver_interface=127.3.217.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.61.keytab
--principal=kudu/127.3.217.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:32975,127.3.217.61:45143,127.3.217.60:35929
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:08:25.674194 17310 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } attempt: 1
I20260924 07:08:25.675256 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.674910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:45143 (local address 127.0.0.1:56050)
0924 07:08:25.675094 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:25.675165 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.61:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 07:08:25.676420 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.676170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:45143 (local address 127.0.0.1:56054)
0924 07:08:25.676305 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:25.676357 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.61:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260924 07:08:25.676559 17310 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.61:45143: Network error: Client connection negotiation failed: client connection to 127.3.217.61:45143: connect: Connection refused (error 111)
W20260924 07:08:25.743841 17317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:25.744107 17317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:25.744148 17317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 07:08:25.744714 17310 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } attempt: 2
I20260924 07:08:25.745554 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.745258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:45143 (local address 127.0.0.1:56066)
0924 07:08:25.745410 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:25.745471 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.61:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 07:08:25.746516 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.746272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:45143 (local address 127.0.0.1:56072)
0924 07:08:25.746373 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:25.746420 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.61:45143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 07:08:25.746658 17310 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.61:45143: Network error: Client connection negotiation failed: client connection to 127.3.217.61:45143: connect: Connection refused (error 111)
W20260924 07:08:25.748435 17317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:08:25.748484 17317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:25.748510 17317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:25.748540 17317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:08:25.748569 17317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:08:25.753659 17317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:32975,127.3.217.61:45143,127.3.217.60:35929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.61.keytab
--principal=kudu/127.3.217.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.61:45143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.217.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:25.754681 17317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:25.755640 17317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:25.764149 17322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:25.764166 17325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:25.764170 17323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:25.764742 17317 server_base.cc:1034] running on GCE node
I20260924 07:08:25.765466 17317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:25.766567 17317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:25.767781 17317 hybrid_clock.cc:648] HybridClock initialized: now 1790233705767735 us; error 48 us; skew 500 ppm
Sep 24 07:08:25 dist-test-slave-2p7j krb5kdc[17229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233705, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:25.771140 17317 init.cc:377] Logged in from keytab as kudu/127.3.217.61@KRBTEST.COM (short username kudu)
I20260924 07:08:25.772037 17317 webserver.cc:460] Webserver started at http://127.3.217.61:44509/ using document root <none> and password file <none>
I20260924 07:08:25.772539 17317 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:25.772604 17317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:08:25.772776 17317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:08:25.774660 17317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-1/data/instance:
uuid: "52e6ced76ffa4bc397467a0323743cc9"
format_stamp: "Formatted at 2026-09-24 07:08:25 on dist-test-slave-2p7j"
server_key: "f8ed62f87914539a3c8363d63614471b"
server_key_iv: "b90adff103d70df4415df6067e44c94f"
server_key_version: "encryptionkey@0"
I20260924 07:08:25.775228 17317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-1/wal/instance:
uuid: "52e6ced76ffa4bc397467a0323743cc9"
format_stamp: "Formatted at 2026-09-24 07:08:25 on dist-test-slave-2p7j"
server_key: "f8ed62f87914539a3c8363d63614471b"
server_key_iv: "b90adff103d70df4415df6067e44c94f"
server_key_version: "encryptionkey@0"
I20260924 07:08:25.778662 17317 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 07:08:25.781045 17332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:25.782435 17317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 07:08:25.782593 17317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-1/wal
uuid: "52e6ced76ffa4bc397467a0323743cc9"
format_stamp: "Formatted at 2026-09-24 07:08:25 on dist-test-slave-2p7j"
server_key: "f8ed62f87914539a3c8363d63614471b"
server_key_iv: "b90adff103d70df4415df6067e44c94f"
server_key_version: "encryptionkey@0"
I20260924 07:08:25.782685 17317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:25.796516 17317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:25.799337 17317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:25.799554 17317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:25.806391 17317 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.61:45143
I20260924 07:08:25.806414 17384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.61:45143 every 8 connection(s)
I20260924 07:08:25.807381 17317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-1/data/info.pb
I20260924 07:08:25.810974 17385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:08:25.815793 17385 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.62" port: 32975 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:08:25.817317 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 17317
I20260924 07:08:25.817438 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-1/wal/instance
I20260924 07:08:25.817698 3940 external_mini_cluster.cc:1428] Setting key d2c748d2533e79b016a949fc1c3e6d31
WARNING: no policy specified for kudu/127.3.217.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.60@KRBTEST.COM" created.
Sep 24 07:08:25 dist-test-slave-2p7j krb5kdc[17229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233705, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.61@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:08:25.834240 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.817723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.3.217.62:32975)
0924 07:08:25.817889 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:25.817894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:25.818729 (+ 835us) server_negotiation.cc:407] Connection header received
0924 07:08:25.819688 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:25.819692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:25.819746 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:25.819850 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:25.821950 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:25.822680 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:25.824901 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:25.825184 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:25.825282 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:25.830136 (+ 4854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:25.830140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:25.830143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:25.830171 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:25.831606 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:25.832257 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:25.832260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:25.832261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:25.832339 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:25.832713 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:25.832716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:25.832717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:25.832930 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:25.833019 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:25.833839 (+ 820us) server_negotiation.cc:299] Negotiation successful
0924 07:08:25.834028 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 07:08:25.834956 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.818093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:32975 (local address 127.0.0.1:34726)
0924 07:08:25.818549 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:25.818615 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:25.819453 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:25.820572 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:25.820580 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:25.821028 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:25.821664 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:25.821685 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:25.822872 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:25.822876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:25.824594 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:25.824617 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:25.825375 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:25.825381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:25.825652 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:25.826530 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:25.826551 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:25.829939 (+ 3388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:25.831715 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:25.831721 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:25.831734 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:25.832059 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:25.832426 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:25.832428 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:25.832430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:25.832592 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:25.833045 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:25.833051 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:25.833367 (+ 316us) client_negotiation.cc:769] Sending connection context
0924 07:08:25.833588 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 07:08:25.833931 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":118,"threads_started":1}
I20260924 07:08:25.836405 17385 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:08:25.837790 17310 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } attempt: 3
Sep 24 07:08:25 dist-test-slave-2p7j krb5kdc[17229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233705, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.61@KRBTEST.COM for kudu/127.3.217.61@KRBTEST.COM
Sep 24 07:08:25 dist-test-slave-2p7j krb5kdc[17229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233705, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for kudu/127.3.217.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.60.keytab.
Entry for principal kudu/127.3.217.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.60.keytab.
I20260924 07:08:25.847486 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.60:35929
--webserver_interface=127.3.217.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.60.keytab
--principal=kudu/127.3.217.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:32975,127.3.217.61:45143,127.3.217.60:35929
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:08:25.848160 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.836874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:45143 (local address 127.0.0.1:56086)
0924 07:08:25.836990 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:25.837002 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:25.837064 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:25.837703 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:25.837706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:25.837835 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:25.837918 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:25.837924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:25.839484 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:25.839488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:25.840289 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:25.840301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:25.841016 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:25.841019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:25.841119 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:25.841575 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:25.841589 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:25.843302 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:25.845751 (+ 2449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:25.845757 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:25.845759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:25.846186 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:25.846607 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:25.846611 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:25.846613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:25.846727 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:25.847298 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:25.847300 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:25.847453 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 07:08:25.847686 (+ 233us) client_negotiation.cc:240] Negotiation successful
0924 07:08:25.848002 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 07:08:25.848436 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.836989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56086 (local address 127.3.217.61:45143)
0924 07:08:25.837360 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:25.837369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:25.837395 (+ 26us) server_negotiation.cc:407] Connection header received
0924 07:08:25.837427 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:25.837431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:25.837564 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:25.837667 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:25.838185 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:25.839349 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:25.840417 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:25.840881 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:25.841002 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:25.843484 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:25.843494 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:25.843530 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:25.843553 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:25.845532 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:25.846403 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:25.846408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:25.846409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:25.846504 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:25.846827 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:25.846830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:25.846832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:25.847146 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:25.847263 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:25.848046 (+ 783us) server_negotiation.cc:299] Negotiation successful
0924 07:08:25.848264 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":144,"threads_started":1}
I20260924 07:08:25.849951 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.838529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:45143 (local address 127.0.0.1:56092)
0924 07:08:25.838677 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:25.838694 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:25.838771 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:25.839788 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:25.839791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:25.839972 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:25.840521 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:25.840530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:25.841470 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:25.841474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:25.842284 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:25.842296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:25.842942 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:25.842946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:25.843036 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:25.843417 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:25.843431 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:25.844990 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:25.848432 (+ 3442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:25.848435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:25.848437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:25.848699 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:25.849007 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:25.849010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:25.849012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:25.849095 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:25.849486 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:25.849489 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:25.849581 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 07:08:25.849664 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:08:25.849790 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 07:08:25.849963 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.838561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56092 (local address 127.3.217.61:45143)
0924 07:08:25.839599 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:25.839606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:25.839616 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:08:25.839641 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:25.839642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:25.839673 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:25.839739 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:25.840629 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:25.841297 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:25.842426 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:25.842793 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:25.842897 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:25.845131 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:25.845134 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:25.845136 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:25.845156 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:25.848314 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:25.848832 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:25.848835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:25.848837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:25.848916 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:25.849193 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:25.849196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:25.849197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:25.849374 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:25.849492 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:25.849726 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 07:08:25.849814 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":982,"thread_start_us":104,"threads_started":1}
I20260924 07:08:25.850603 17385 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:08:25.851549 17310 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:08:25.851722 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.851185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:35929 (local address 127.0.0.1:56070)
0924 07:08:25.851385 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:25.851623 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260924 07:08:25.852226 17334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.217.60:35929: connect: Connection refused (error 111)
I20260924 07:08:25.852346 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.852002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:35929 (local address 127.0.0.1:56082)
0924 07:08:25.852163 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:25.852221 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 07:08:25.853179 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.852934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:35929 (local address 127.0.0.1:56106)
0924 07:08:25.853055 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:25.853114 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 07:08:25.853336 17385 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.60:35929: Network error: Client connection negotiation failed: client connection to 127.3.217.60:35929: connect: Connection refused (error 111)
I20260924 07:08:25.853717 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.852949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:35929 (local address 127.0.0.1:56094)
0924 07:08:25.853531 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:25.853606 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":144,"threads_started":1}
W20260924 07:08:25.853950 17310 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.60:35929: Network error: Client connection negotiation failed: client connection to 127.3.217.60:35929: connect: Connection refused (error 111)
I20260924 07:08:25.885149 17310 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } attempt: 1
I20260924 07:08:25.886076 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.885714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:35929 (local address 127.0.0.1:56112)
0924 07:08:25.885880 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:25.885984 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 07:08:25.887276 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.886896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:35929 (local address 127.0.0.1:56122)
0924 07:08:25.887072 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:25.887181 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260924 07:08:25.887471 17310 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.60:35929: Network error: Client connection negotiation failed: client connection to 127.3.217.60:35929: connect: Connection refused (error 111)
I20260924 07:08:25.902642 17385 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } attempt: 1
I20260924 07:08:25.903467 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.903141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:35929 (local address 127.0.0.1:56132)
0924 07:08:25.903308 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:25.903378 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 07:08:25.904219 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.903990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:35929 (local address 127.0.0.1:56136)
0924 07:08:25.904110 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:25.904164 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 07:08:25.904464 17385 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.60:35929: Network error: Client connection negotiation failed: client connection to 127.3.217.60:35929: connect: Connection refused (error 111)
I20260924 07:08:25.962675 17310 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } attempt: 2
I20260924 07:08:25.963742 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.963215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:35929 (local address 127.0.0.1:56138)
0924 07:08:25.963490 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:25.963624 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260924 07:08:25.964596 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.964381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:35929 (local address 127.0.0.1:56142)
0924 07:08:25.964487 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:25.964536 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 07:08:25.964721 17310 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.60:35929: Network error: Client connection negotiation failed: client connection to 127.3.217.60:35929: connect: Connection refused (error 111)
I20260924 07:08:25.972733 17385 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } attempt: 2
I20260924 07:08:25.973614 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.973277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:35929 (local address 127.0.0.1:56150)
0924 07:08:25.973461 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:25.973530 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260924 07:08:25.973595 17392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:25.973917 17392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:25.974000 17392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 07:08:25.974651 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:25.974268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:35929 (local address 127.0.0.1:56160)
0924 07:08:25.974475 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:25.974558 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260924 07:08:25.974853 17385 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.60:35929: Network error: Client connection negotiation failed: client connection to 127.3.217.60:35929: connect: Connection refused (error 111)
W20260924 07:08:25.978952 17392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:08:25.979012 17392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:25.979029 17392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:25.979045 17392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:08:25.979061 17392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:08:25.984433 17392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:32975,127.3.217.61:45143,127.3.217.60:35929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.60.keytab
--principal=kudu/127.3.217.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.60:35929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.217.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:25.985438 17392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:25.986491 17392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:25.995231 17399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:25.995375 17392 server_base.cc:1034] running on GCE node
W20260924 07:08:25.995386 17400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:25.995231 17402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:25.996212 17392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:25.997260 17392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:25.998474 17392 hybrid_clock.cc:648] HybridClock initialized: now 1790233705998467 us; error 47 us; skew 500 ppm
Sep 24 07:08:25 dist-test-slave-2p7j krb5kdc[17229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233705, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:26.001130 17392 init.cc:377] Logged in from keytab as kudu/127.3.217.60@KRBTEST.COM (short username kudu)
I20260924 07:08:26.002102 17392 webserver.cc:460] Webserver started at http://127.3.217.60:36991/ using document root <none> and password file <none>
I20260924 07:08:26.002650 17392 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:26.002714 17392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:08:26.002892 17392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:08:26.004842 17392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-2/data/instance:
uuid: "c87c03beab694faebf880feb43ee4c4d"
format_stamp: "Formatted at 2026-09-24 07:08:25 on dist-test-slave-2p7j"
server_key: "4950736c73dcf7c1afeb9def2a31d1f9"
server_key_iv: "575fff686989c0c1895d4187b8b4a7ea"
server_key_version: "encryptionkey@0"
I20260924 07:08:26.005348 17392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-2/wal/instance:
uuid: "c87c03beab694faebf880feb43ee4c4d"
format_stamp: "Formatted at 2026-09-24 07:08:25 on dist-test-slave-2p7j"
server_key: "4950736c73dcf7c1afeb9def2a31d1f9"
server_key_iv: "575fff686989c0c1895d4187b8b4a7ea"
server_key_version: "encryptionkey@0"
I20260924 07:08:26.008772 17392 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260924 07:08:26.011471 17409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:26.013303 17392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:08:26.013502 17392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-2/wal
uuid: "c87c03beab694faebf880feb43ee4c4d"
format_stamp: "Formatted at 2026-09-24 07:08:25 on dist-test-slave-2p7j"
server_key: "4950736c73dcf7c1afeb9def2a31d1f9"
server_key_iv: "575fff686989c0c1895d4187b8b4a7ea"
server_key_version: "encryptionkey@0"
I20260924 07:08:26.013661 17392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:26.034878 17392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:26.038618 17392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:26.038802 17392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:26.046294 17392 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.60:35929
I20260924 07:08:26.046320 17461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.60:35929 every 8 connection(s)
I20260924 07:08:26.047168 17392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-2/data/info.pb
I20260924 07:08:26.049925 17462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:08:26.053678 17462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.62" port: 32975 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:08:26.055161 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 17392
I20260924 07:08:26.055320 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233541052846-3940-0/minicluster-data/master-2/wal/instance
I20260924 07:08:26.055752 3940 external_mini_cluster.cc:1428] Setting key 637a594659f6ddeb85c1b7c5001bfbd3
I20260924 07:08:26.063928 17310 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } attempt: 3
Sep 24 07:08:26 dist-test-slave-2p7j krb5kdc[17229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233705, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.60@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:08:26.066013 17385 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } attempt: 3
Sep 24 07:08:26 dist-test-slave-2p7j krb5kdc[17229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.60@KRBTEST.COM
Sep 24 07:08:26 dist-test-slave-2p7j krb5kdc[17229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233705, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for kudu/127.3.217.60@KRBTEST.COM
I20260924 07:08:26.073521 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:26.055976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34734 (local address 127.3.217.62:32975)
0924 07:08:26.056128 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:26.056132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:26.057405 (+ 1273us) server_negotiation.cc:407] Connection header received
0924 07:08:26.058706 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:26.058710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:26.058764 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:26.059025 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:26.060196 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.060994 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.062471 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.062960 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.063096 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:26.066565 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:26.066570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:26.066572 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:26.066596 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:26.069344 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:26.070423 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:26.070429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:26.070431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:26.070554 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:26.071046 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:26.071049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:26.071051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:26.071316 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:26.071410 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:26.073166 (+ 1756us) server_negotiation.cc:299] Negotiation successful
0924 07:08:26.073329 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 07:08:26 dist-test-slave-2p7j krb5kdc[17229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233705, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.61@KRBTEST.COM for kudu/127.3.217.60@KRBTEST.COM
I20260924 07:08:26.074293 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:26.056541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:32975 (local address 127.0.0.1:34734)
0924 07:08:26.057220 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:26.057271 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:26.058467 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:26.059076 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:26.059081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:26.059577 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:26.060045 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:26.060059 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.061168 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.061171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:26.062283 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:26.062294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.063188 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.063192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:26.063320 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:26.064292 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:26.064313 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:26.066384 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:26.069761 (+ 3377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:26.069771 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:26.069787 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:26.070234 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:26.070679 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:26.070682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:26.070684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:26.070915 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:26.071462 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:26.071471 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:26.072084 (+ 613us) client_negotiation.cc:769] Sending connection context
0924 07:08:26.072719 (+ 635us) client_negotiation.cc:240] Negotiation successful
0924 07:08:26.073163 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":31,"thread_start_us":253,"threads_started":1}
I20260924 07:08:26.074522 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:26.057657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56172 (local address 127.3.217.60:35929)
0924 07:08:26.058056 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:26.058067 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:26.058103 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:08:26.058492 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:26.058518 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:26.058740 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:26.059108 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:26.060059 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.061271 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.062471 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.063223 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.063321 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:26.067188 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:26.067196 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:26.067210 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:26.067233 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:26.070366 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:26.070959 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:26.070967 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:26.070969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:26.071065 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:26.071378 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:26.071382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:26.071383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:26.071964 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:26.072205 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:26.073169 (+ 964us) server_negotiation.cc:299] Negotiation successful
0924 07:08:26.073330 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":315,"thread_start_us":214,"threads_started":1}
I20260924 07:08:26.076427 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:26.064377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:35929 (local address 127.0.0.1:56180)
0924 07:08:26.064615 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:26.064631 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:26.064718 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:26.065386 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:26.065388 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:26.065556 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:26.065659 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:26.065664 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.066525 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.066528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:26.067583 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:26.067609 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.068484 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.068489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:26.068588 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:26.069287 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:26.069301 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:26.071175 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:26.074770 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:26.074773 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:26.074776 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:26.075070 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:26.075354 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:26.075356 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:26.075358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:26.075439 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:26.075889 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:26.075892 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:26.076054 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 07:08:26.076131 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 07:08:26.076267 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260924 07:08:26.076925 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:26.064878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56180 (local address 127.3.217.60:35929)
0924 07:08:26.065196 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:26.065202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:26.065213 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:08:26.065240 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:26.065242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:26.065274 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:26.065348 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:26.065763 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.066407 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.067796 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.068340 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.068432 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:26.071316 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:26.071320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:26.071323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:26.071345 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:26.074651 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:26.075181 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:26.075184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:26.075185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:26.075264 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:26.075572 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:26.075574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:26.075576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:26.075793 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:26.076589 (+ 796us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:26.076661 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 07:08:26.076776 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":149,"threads_started":1}
I20260924 07:08:26.076995 17462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:08:26.078131 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:26.066386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:35929 (local address 127.0.0.1:56186)
0924 07:08:26.066551 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:26.066578 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:26.066646 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:26.068208 (+ 1562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:26.068212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:26.068414 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:26.068532 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:26.068539 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.069928 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.070075 (+ 147us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:26.071022 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:26.071034 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.072180 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.072185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:26.072273 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:26.072666 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:26.072679 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:26.073951 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:26.075887 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:26.075890 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:26.075892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:26.076147 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:26.076431 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:26.076434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:26.076436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:26.076554 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:26.077661 (+ 1107us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:26.077664 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:26.077763 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 07:08:26.077851 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 07:08:26.077983 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260924 07:08:26.079080 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:26.067034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56186 (local address 127.3.217.60:35929)
0924 07:08:26.067686 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:26.067713 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:26.067730 (+ 17us) server_negotiation.cc:407] Connection header received
0924 07:08:26.067779 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:26.067781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:26.068039 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:26.068146 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:26.068806 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.069796 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.071170 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.072040 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.072442 (+ 402us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:26.074093 (+ 1651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:26.074097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:26.074100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:26.074124 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:26.075768 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:26.076266 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:26.076268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:26.076270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:26.076346 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:26.076891 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:26.076894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:26.076895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:26.077546 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:26.078737 (+ 1191us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:26.078808 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 07:08:26.078912 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":329,"threads_started":1}
I20260924 07:08:26.080437 17310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90cda92a4f4e4e29812187820864d26a: Bootstrap starting.
I20260924 07:08:26.083281 17385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e6ced76ffa4bc397467a0323743cc9: Bootstrap starting.
Sep 24 07:08:26 dist-test-slave-2p7j krb5kdc[17229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233705, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.60@KRBTEST.COM for kudu/127.3.217.61@KRBTEST.COM
I20260924 07:08:26.083587 17310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90cda92a4f4e4e29812187820864d26a: Neither blocks nor log segments found. Creating new log.
Sep 24 07:08:26 dist-test-slave-2p7j krb5kdc[17229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:08:26.085007 17310 log.cc:824] T 00000000000000000000000000000000 P 90cda92a4f4e4e29812187820864d26a: Log is configured to *not* fsync() on all Append() calls
I20260924 07:08:26.087020 17385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52e6ced76ffa4bc397467a0323743cc9: Neither blocks nor log segments found. Creating new log.
I20260924 07:08:26.087872 17385 log.cc:824] T 00000000000000000000000000000000 P 52e6ced76ffa4bc397467a0323743cc9: Log is configured to *not* fsync() on all Append() calls
I20260924 07:08:26.087936 17310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90cda92a4f4e4e29812187820864d26a: No bootstrap required, opened a new log
I20260924 07:08:26.089810 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:26.077398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:45143 (local address 127.0.0.1:56108)
0924 07:08:26.077502 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:26.077515 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:26.077581 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:26.078497 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:26.078500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:26.078642 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:26.078741 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:26.078747 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.080099 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.080102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:26.080829 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:26.080839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.081541 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.081545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:26.081649 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:26.082045 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:26.082058 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:26.084564 (+ 2506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:26.088187 (+ 3623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:26.088190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:26.088192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:26.088429 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:26.088723 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:26.088726 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:26.088727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:26.088812 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:26.089302 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:26.089305 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:26.089404 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 07:08:26.089501 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 07:08:26.089642 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 07:08:26.090289 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:26.077619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56108 (local address 127.3.217.61:45143)
0924 07:08:26.078298 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:26.078302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:26.078314 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:08:26.078346 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:26.078349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:26.078386 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:26.078466 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:26.078864 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.079796 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.080957 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.081294 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.081363 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:26.084810 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:26.084816 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:26.084819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:26.084864 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:26.088063 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:26.088542 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:26.088545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:26.088547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:26.088631 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:26.088919 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:26.088922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:26.088924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:26.089176 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:26.089260 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:26.090033 (+ 773us) server_negotiation.cc:299] Negotiation successful
0924 07:08:26.090153 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617}
I20260924 07:08:26.090890 17462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:08:26.091267 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:26.077262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34736 (local address 127.3.217.62:32975)
0924 07:08:26.077388 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:26.077392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:26.077876 (+ 484us) server_negotiation.cc:407] Connection header received
0924 07:08:26.077965 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:26.077968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:26.078019 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:26.078092 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:26.078791 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.079597 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.080508 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.080896 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.080990 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:26.085284 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:26.085288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:26.085291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:26.085317 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:26.087282 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:26.087789 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:26.087792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:26.087794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:26.087879 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:26.088882 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:26.088887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:26.088890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:26.089192 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:26.089436 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:26.090644 (+ 1208us) server_negotiation.cc:299] Negotiation successful
0924 07:08:26.091034 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:08:26.091617 17385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e6ced76ffa4bc397467a0323743cc9: No bootstrap required, opened a new log
I20260924 07:08:26.092654 17310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90cda92a4f4e4e29812187820864d26a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90cda92a4f4e4e29812187820864d26a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 32975 } } peers { permanent_uuid: "52e6ced76ffa4bc397467a0323743cc9" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } } peers { permanent_uuid: "c87c03beab694faebf880feb43ee4c4d" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } }
I20260924 07:08:26.092854 17310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90cda92a4f4e4e29812187820864d26a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:08:26.092904 17310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90cda92a4f4e4e29812187820864d26a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90cda92a4f4e4e29812187820864d26a, State: Initialized, Role: FOLLOWER
I20260924 07:08:26.093319 17310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90cda92a4f4e4e29812187820864d26a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90cda92a4f4e4e29812187820864d26a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 32975 } } peers { permanent_uuid: "52e6ced76ffa4bc397467a0323743cc9" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } } peers { permanent_uuid: "c87c03beab694faebf880feb43ee4c4d" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } }
I20260924 07:08:26.093771 17471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90cda92a4f4e4e29812187820864d26a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90cda92a4f4e4e29812187820864d26a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 32975 } } peers { permanent_uuid: "52e6ced76ffa4bc397467a0323743cc9" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } } peers { permanent_uuid: "c87c03beab694faebf880feb43ee4c4d" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } } }
I20260924 07:08:26.093968 17471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90cda92a4f4e4e29812187820864d26a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:08:26.094547 17310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90cda92a4f4e4e29812187820864d26a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:08:26.095438 17385 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52e6ced76ffa4bc397467a0323743cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90cda92a4f4e4e29812187820864d26a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 32975 } } peers { permanent_uuid: "52e6ced76ffa4bc397467a0323743cc9" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } } peers { permanent_uuid: "c87c03beab694faebf880feb43ee4c4d" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } }
I20260924 07:08:26.095818 17385 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52e6ced76ffa4bc397467a0323743cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:08:26.095872 17385 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52e6ced76ffa4bc397467a0323743cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52e6ced76ffa4bc397467a0323743cc9, State: Initialized, Role: FOLLOWER
I20260924 07:08:26.096318 17385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52e6ced76ffa4bc397467a0323743cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90cda92a4f4e4e29812187820864d26a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 32975 } } peers { permanent_uuid: "52e6ced76ffa4bc397467a0323743cc9" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } } peers { permanent_uuid: "c87c03beab694faebf880feb43ee4c4d" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } }
Sep 24 07:08:26 dist-test-slave-2p7j krb5kdc[17229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233705, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.60@KRBTEST.COM for kudu/127.3.217.60@KRBTEST.COM
I20260924 07:08:26.097569 17385 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52e6ced76ffa4bc397467a0323743cc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:08:26.100454 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:26.091607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56202 (local address 127.3.217.60:35929)
0924 07:08:26.091721 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:26.091725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:26.092098 (+ 373us) server_negotiation.cc:407] Connection header received
0924 07:08:26.092189 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:26.092192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:26.092238 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:26.092320 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:26.092702 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.093461 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.094498 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.094825 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.094890 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:26.097171 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:26.097175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:26.097178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:26.097199 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:26.098781 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:26.099243 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:26.099246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:26.099248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:26.099317 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:26.099594 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:26.099597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:26.099599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:26.099796 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:26.099861 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:26.100189 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 07:08:26.100306 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 07:08:26.100823 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:26.091328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:35929 (local address 127.0.0.1:56202)
0924 07:08:26.092015 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:26.092029 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:26.092113 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:26.092344 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:26.092346 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:26.092502 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:26.092590 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:26.092596 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.093572 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.093574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:26.094383 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:26.094393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.095028 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.095031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:26.095100 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:26.095596 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:26.095609 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:26.097045 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:26.098899 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:26.098902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:26.098904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:26.099142 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:26.099405 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:26.099408 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:26.099410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:26.099479 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:26.099939 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:26.099942 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:26.100025 (+ 83us) client_negotiation.cc:769] Sending connection context
0924 07:08:26.100571 (+ 546us) client_negotiation.cc:240] Negotiation successful
0924 07:08:26.100690 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620}
I20260924 07:08:26.099174 17242 master_runner.cc:186] Error getting master registration for 127.3.217.61:45143: OK, instance_id { permanent_uuid: "52e6ced76ffa4bc397467a0323743cc9" instance_seqno: 1790233705804455 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:08:26.102312 17242 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:08:26.103999 17462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d: Bootstrap starting.
W20260924 07:08:26.104609 17494 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 52e6ced76ffa4bc397467a0323743cc9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:08:26.104673 17494 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 07:08:26.106952 17482 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 90cda92a4f4e4e29812187820864d26a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:08:26.107009 17482 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:08:26.107098 17462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d: Neither blocks nor log segments found. Creating new log.
I20260924 07:08:26.107981 17462 log.cc:824] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:08:26.104981 17317 master_runner.cc:186] Error getting master registration for 127.3.217.60:35929: OK, instance_id { permanent_uuid: "c87c03beab694faebf880feb43ee4c4d" instance_seqno: 1790233706044227 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:08:26.109078 17317 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 24 07:08:26 dist-test-slave-2p7j krb5kdc[17229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.61@KRBTEST.COM
I20260924 07:08:26.111742 17483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e6ced76ffa4bc397467a0323743cc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90cda92a4f4e4e29812187820864d26a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 32975 } } peers { permanent_uuid: "52e6ced76ffa4bc397467a0323743cc9" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } } peers { permanent_uuid: "c87c03beab694faebf880feb43ee4c4d" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } } }
I20260924 07:08:26.111940 17483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e6ced76ffa4bc397467a0323743cc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:08:26.115314 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:26.096902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56112 (local address 127.3.217.61:45143)
0924 07:08:26.097008 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:26.097011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:26.097189 (+ 178us) server_negotiation.cc:407] Connection header received
0924 07:08:26.097356 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:26.097359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:26.097402 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:26.097450 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:26.098319 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.098987 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.100041 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.100354 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.100417 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:26.111166 (+ 10749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:26.111171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:26.111174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:26.111204 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:26.113264 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:26.113774 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:26.113778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:26.113780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:26.113862 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:26.114252 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:26.114256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:26.114258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:26.114498 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:26.114660 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:26.115022 (+ 362us) server_negotiation.cc:299] Negotiation successful
0924 07:08:26.115159 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 07:08:26.118901 17462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d: No bootstrap required, opened a new log
I20260924 07:08:26.119045 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:08:26.123823 17462 raft_consensus.cc:348] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90cda92a4f4e4e29812187820864d26a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 32975 } } peers { permanent_uuid: "52e6ced76ffa4bc397467a0323743cc9" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } } peers { permanent_uuid: "c87c03beab694faebf880feb43ee4c4d" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } }
I20260924 07:08:26.124111 17462 raft_consensus.cc:374] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:08:26.124174 17462 raft_consensus.cc:729] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87c03beab694faebf880feb43ee4c4d, State: Initialized, Role: FOLLOWER
I20260924 07:08:26.124627 17462 consensus_queue.cc:260] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90cda92a4f4e4e29812187820864d26a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 32975 } } peers { permanent_uuid: "52e6ced76ffa4bc397467a0323743cc9" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } } peers { permanent_uuid: "c87c03beab694faebf880feb43ee4c4d" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } }
I20260924 07:08:26.125917 17462 sys_catalog.cc:564] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:08:26.128994 17503 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90cda92a4f4e4e29812187820864d26a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 32975 } } peers { permanent_uuid: "52e6ced76ffa4bc397467a0323743cc9" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } } peers { permanent_uuid: "c87c03beab694faebf880feb43ee4c4d" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } } }
I20260924 07:08:26.129142 17503 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [sys.catalog]: This master's current role is: FOLLOWER
W20260924 07:08:26.131820 17516 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:08:26.131934 17516 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:08:26.135594 17392 master_runner.cc:263] No leader master found from master c87c03beab694faebf880feb43ee4c4d
I20260924 07:08:26.135660 17392 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:08:26.141623 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:26.123912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34746 (local address 127.3.217.62:32975)
0924 07:08:26.124030 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:26.124034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:26.131735 (+ 7701us) server_negotiation.cc:407] Connection header received
0924 07:08:26.131923 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:26.131926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:26.132156 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:26.132285 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:26.133427 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.134216 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.135528 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.135879 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.135961 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:26.137894 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:26.137898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:26.137902 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:26.137929 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:26.139700 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:26.140207 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:26.140211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:26.140214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:26.140300 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:26.140609 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:26.140613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:26.140615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:26.140833 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:26.140962 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:26.141328 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 07:08:26.141450 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:08:26.143139 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:26.124414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56206 (local address 127.3.217.60:35929)
0924 07:08:26.124517 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:26.124521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:26.133655 (+ 9134us) server_negotiation.cc:407] Connection header received
0924 07:08:26.133847 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:26.133850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:26.133903 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:26.133955 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:26.135465 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.136062 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.136909 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.137244 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.137334 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:26.139641 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:26.139645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:26.139648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:26.139672 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:26.141201 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:26.141650 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:26.141654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:26.141655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:26.141731 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:26.142249 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:26.142252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:26.142254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:26.142447 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:26.142567 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:26.142876 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 07:08:26.142995 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 07:08:26.151561 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:26.124717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56122 (local address 127.3.217.61:45143)
0924 07:08:26.124812 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:26.124816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:26.143386 (+ 18570us) server_negotiation.cc:407] Connection header received
0924 07:08:26.143595 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:26.143599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:26.143660 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:26.143722 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:26.144574 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.145309 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.146361 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:26.146687 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:26.146786 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:26.148296 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:26.148300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:26.148303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:26.148327 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:26.149908 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:26.150357 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:26.150361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:26.150363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:26.150443 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:26.150710 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:26.150713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:26.150715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:26.150916 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:26.151052 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:26.151279 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 07:08:26.151385 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 07:08:26.217819 17503 raft_consensus.cc:482] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:08:26.218034 17503 raft_consensus.cc:504] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90cda92a4f4e4e29812187820864d26a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 32975 } } peers { permanent_uuid: "52e6ced76ffa4bc397467a0323743cc9" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } } peers { permanent_uuid: "c87c03beab694faebf880feb43ee4c4d" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } }
I20260924 07:08:26.219095 17503 leader_election.cc:288] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90cda92a4f4e4e29812187820864d26a (127.3.217.62:32975), 52e6ced76ffa4bc397467a0323743cc9 (127.3.217.61:45143)
I20260924 07:08:26.219882 17284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c87c03beab694faebf880feb43ee4c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90cda92a4f4e4e29812187820864d26a" is_pre_election: true
I20260924 07:08:26.220175 17284 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 90cda92a4f4e4e29812187820864d26a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87c03beab694faebf880feb43ee4c4d in term 0.
I20260924 07:08:26.220863 17359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c87c03beab694faebf880feb43ee4c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52e6ced76ffa4bc397467a0323743cc9" is_pre_election: true
I20260924 07:08:26.221130 17359 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 52e6ced76ffa4bc397467a0323743cc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87c03beab694faebf880feb43ee4c4d in term 0.
I20260924 07:08:26.221751 17412 leader_election.cc:302] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52e6ced76ffa4bc397467a0323743cc9, c87c03beab694faebf880feb43ee4c4d; no voters:
I20260924 07:08:26.222453 17503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:08:26.222546 17503 raft_consensus.cc:482] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:08:26.222589 17503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:08:26.224028 17503 raft_consensus.cc:504] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90cda92a4f4e4e29812187820864d26a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 32975 } } peers { permanent_uuid: "52e6ced76ffa4bc397467a0323743cc9" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } } peers { permanent_uuid: "c87c03beab694faebf880feb43ee4c4d" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } }
I20260924 07:08:26.224391 17503 leader_election.cc:288] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [CANDIDATE]: Term 1 election: Requested vote from peers 90cda92a4f4e4e29812187820864d26a (127.3.217.62:32975), 52e6ced76ffa4bc397467a0323743cc9 (127.3.217.61:45143)
I20260924 07:08:26.225023 17359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c87c03beab694faebf880feb43ee4c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52e6ced76ffa4bc397467a0323743cc9"
I20260924 07:08:26.225137 17359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52e6ced76ffa4bc397467a0323743cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:08:26.226351 17359 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 52e6ced76ffa4bc397467a0323743cc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87c03beab694faebf880feb43ee4c4d in term 1.
I20260924 07:08:26.226873 17412 leader_election.cc:302] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52e6ced76ffa4bc397467a0323743cc9, c87c03beab694faebf880feb43ee4c4d; no voters:
I20260924 07:08:26.227110 17284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c87c03beab694faebf880feb43ee4c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90cda92a4f4e4e29812187820864d26a"
I20260924 07:08:26.227221 17284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90cda92a4f4e4e29812187820864d26a [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:08:26.228327 17284 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 90cda92a4f4e4e29812187820864d26a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87c03beab694faebf880feb43ee4c4d in term 1.
I20260924 07:08:26.228837 17503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:08:26.229090 17503 raft_consensus.cc:686] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [term 1 LEADER]: Becoming Leader. State: Replica: c87c03beab694faebf880feb43ee4c4d, State: Running, Role: LEADER
I20260924 07:08:26.231438 17503 consensus_queue.cc:237] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90cda92a4f4e4e29812187820864d26a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 32975 } } peers { permanent_uuid: "52e6ced76ffa4bc397467a0323743cc9" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } } peers { permanent_uuid: "c87c03beab694faebf880feb43ee4c4d" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } }
I20260924 07:08:26.234714 17503 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c87c03beab694faebf880feb43ee4c4d. Latest consensus state: current_term: 1 leader_uuid: "c87c03beab694faebf880feb43ee4c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90cda92a4f4e4e29812187820864d26a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 32975 } } peers { permanent_uuid: "52e6ced76ffa4bc397467a0323743cc9" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } } peers { permanent_uuid: "c87c03beab694faebf880feb43ee4c4d" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } } }
I20260924 07:08:26.234884 17503 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [sys.catalog]: This master's current role is: LEADER
I20260924 07:08:26.241882 17520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:08:26.245342 17520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:08:26.252238 17359 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 52e6ced76ffa4bc397467a0323743cc9 [term 1 FOLLOWER]: Refusing update from remote peer c87c03beab694faebf880feb43ee4c4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:08:26.253347 17284 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90cda92a4f4e4e29812187820864d26a [term 1 FOLLOWER]: Refusing update from remote peer c87c03beab694faebf880feb43ee4c4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:08:26.254311 17503 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "52e6ced76ffa4bc397467a0323743cc9" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:08:26.254737 17503 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90cda92a4f4e4e29812187820864d26a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:08:26.264505 17483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e6ced76ffa4bc397467a0323743cc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c87c03beab694faebf880feb43ee4c4d. Latest consensus state: current_term: 1 leader_uuid: "c87c03beab694faebf880feb43ee4c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90cda92a4f4e4e29812187820864d26a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 32975 } } peers { permanent_uuid: "52e6ced76ffa4bc397467a0323743cc9" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } } peers { permanent_uuid: "c87c03beab694faebf880feb43ee4c4d" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } } }
I20260924 07:08:26.264665 17483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e6ced76ffa4bc397467a0323743cc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:08:26.266757 17471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90cda92a4f4e4e29812187820864d26a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c87c03beab694faebf880feb43ee4c4d. Latest consensus state: current_term: 1 leader_uuid: "c87c03beab694faebf880feb43ee4c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90cda92a4f4e4e29812187820864d26a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 32975 } } peers { permanent_uuid: "52e6ced76ffa4bc397467a0323743cc9" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } } peers { permanent_uuid: "c87c03beab694faebf880feb43ee4c4d" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } } }
I20260924 07:08:26.266901 17471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90cda92a4f4e4e29812187820864d26a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:08:26.272771 17483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e6ced76ffa4bc397467a0323743cc9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c87c03beab694faebf880feb43ee4c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90cda92a4f4e4e29812187820864d26a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 32975 } } peers { permanent_uuid: "52e6ced76ffa4bc397467a0323743cc9" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } } peers { permanent_uuid: "c87c03beab694faebf880feb43ee4c4d" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } } }
I20260924 07:08:26.272905 17483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e6ced76ffa4bc397467a0323743cc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:08:26.273830 17503 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c87c03beab694faebf880feb43ee4c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90cda92a4f4e4e29812187820864d26a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 32975 } } peers { permanent_uuid: "52e6ced76ffa4bc397467a0323743cc9" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } } peers { permanent_uuid: "c87c03beab694faebf880feb43ee4c4d" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } } }
I20260924 07:08:26.273957 17503 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [sys.catalog]: This master's current role is: LEADER
I20260924 07:08:26.274333 17521 mvcc.cc:205] Tried to move back new op lower bound from 7332797260801085440 to 7332797260729012224. Current Snapshot: MvccSnapshot[applied={T|T < 7332797260801085440}]
I20260924 07:08:26.275777 17523 mvcc.cc:205] Tried to move back new op lower bound from 7332797260801085440 to 7332797260729012224. Current Snapshot: MvccSnapshot[applied={T|T < 7332797260801085440}]
I20260924 07:08:26.277524 17518 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c87c03beab694faebf880feb43ee4c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90cda92a4f4e4e29812187820864d26a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 32975 } } peers { permanent_uuid: "52e6ced76ffa4bc397467a0323743cc9" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } } peers { permanent_uuid: "c87c03beab694faebf880feb43ee4c4d" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } } }
I20260924 07:08:26.277658 17518 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d [sys.catalog]: This master's current role is: LEADER
I20260924 07:08:26.279074 17471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90cda92a4f4e4e29812187820864d26a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c87c03beab694faebf880feb43ee4c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90cda92a4f4e4e29812187820864d26a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 32975 } } peers { permanent_uuid: "52e6ced76ffa4bc397467a0323743cc9" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 45143 } } peers { permanent_uuid: "c87c03beab694faebf880feb43ee4c4d" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 35929 } } }
I20260924 07:08:26.279196 17471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90cda92a4f4e4e29812187820864d26a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:08:26.290542 17520 catalog_manager.cc:1347] Generated new cluster ID: 66e6f80bc2934d2989f1b3d5d8e637f3
I20260924 07:08:26.290650 17520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:08:26.321050 17520 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:08:26.322398 17520 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:08:26.334327 17520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c87c03beab694faebf880feb43ee4c4d: Generated new TSK 0
I20260924 07:08:26.335376 17520 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:08:27.106498 17494 catalog_manager.cc:1259] Loaded cluster ID: 66e6f80bc2934d2989f1b3d5d8e637f3
I20260924 07:08:27.106611 17494 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 52e6ced76ffa4bc397467a0323743cc9: loading cluster ID for follower catalog manager: success
I20260924 07:08:27.108397 17482 catalog_manager.cc:1259] Loaded cluster ID: 66e6f80bc2934d2989f1b3d5d8e637f3
I20260924 07:08:27.108493 17482 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 90cda92a4f4e4e29812187820864d26a: loading cluster ID for follower catalog manager: success
I20260924 07:08:27.108945 17494 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 52e6ced76ffa4bc397467a0323743cc9: acquiring CA information for follower catalog manager: success
I20260924 07:08:27.110323 17482 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 90cda92a4f4e4e29812187820864d26a: acquiring CA information for follower catalog manager: success
I20260924 07:08:27.110616 17494 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 52e6ced76ffa4bc397467a0323743cc9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:08:27.111610 17482 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 90cda92a4f4e4e29812187820864d26a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:08:27.412019 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 17242
I20260924 07:08:27.418205 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 17317
I20260924 07:08:27.425345 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 17392
2026-09-24T07:08:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3157 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:08:27 dist-test-slave-2p7j krb5kdc[17535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:08:27 dist-test-slave-2p7j krb5kdc[17535](info): set up 2 sockets
Sep 24 07:08:27 dist-test-slave-2p7j krb5kdc[17535](info): commencing operation
krb5kdc: starting...
W20260924 07:08:28.408088 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:08:28 dist-test-slave-2p7j krb5kdc[17535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:08:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:08:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:08:28.540607 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:38265
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:43501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:38265
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:08:28.661496 17548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:28.661734 17548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:28.661774 17548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:28.665699 17548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:08:28.665747 17548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:28.665766 17548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:28.665779 17548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:08:28.665795 17548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:08:28.670818 17548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:43501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:38265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:38265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:28.671799 17548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:28.673086 17548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:28.682540 17554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:28.682541 17553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:28.682950 17556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:28.683282 17548 server_base.cc:1034] running on GCE node
I20260924 07:08:28.683822 17548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:28.685444 17548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:28.686756 17548 hybrid_clock.cc:648] HybridClock initialized: now 1790233708686718 us; error 39 us; skew 500 ppm
Sep 24 07:08:28 dist-test-slave-2p7j krb5kdc[17535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233708, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:28.690891 17548 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:08:28.692345 17548 webserver.cc:460] Webserver started at http://127.3.217.62:37047/ using document root <none> and password file <none>
I20260924 07:08:28.692968 17548 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:28.693037 17548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:08:28.693228 17548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:08:28.695282 17548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "a68e296c1fdb43b2863b077ed098de0c"
format_stamp: "Formatted at 2026-09-24 07:08:28 on dist-test-slave-2p7j"
server_key: "38149438cda301c082401bd41d4151ff"
server_key_iv: "9da35d0b0d41664b9d93e01e866adf3d"
server_key_version: "encryptionkey@0"
I20260924 07:08:28.695850 17548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "a68e296c1fdb43b2863b077ed098de0c"
format_stamp: "Formatted at 2026-09-24 07:08:28 on dist-test-slave-2p7j"
server_key: "38149438cda301c082401bd41d4151ff"
server_key_iv: "9da35d0b0d41664b9d93e01e866adf3d"
server_key_version: "encryptionkey@0"
I20260924 07:08:28.700167 17548 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:08:28.702903 17563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:28.704284 17548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:08:28.704422 17548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "a68e296c1fdb43b2863b077ed098de0c"
format_stamp: "Formatted at 2026-09-24 07:08:28 on dist-test-slave-2p7j"
server_key: "38149438cda301c082401bd41d4151ff"
server_key_iv: "9da35d0b0d41664b9d93e01e866adf3d"
server_key_version: "encryptionkey@0"
I20260924 07:08:28.704520 17548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:28.729362 17548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:28.732498 17548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:28.732689 17548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:28.739249 17548 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:38265
I20260924 07:08:28.739303 17615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:38265 every 8 connection(s)
I20260924 07:08:28.740186 17548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:08:28.742743 17616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:08:28.747870 17616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a68e296c1fdb43b2863b077ed098de0c: Bootstrap starting.
I20260924 07:08:28.748749 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 17548
I20260924 07:08:28.748873 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:08:28.749176 3940 external_mini_cluster.cc:1428] Setting key 123ebe12e7892beaa86a31fe376b7bd5
I20260924 07:08:28.750489 17616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a68e296c1fdb43b2863b077ed098de0c: Neither blocks nor log segments found. Creating new log.
I20260924 07:08:28.751209 17616 log.cc:824] T 00000000000000000000000000000000 P a68e296c1fdb43b2863b077ed098de0c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:08:28.754199 17616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a68e296c1fdb43b2863b077ed098de0c: No bootstrap required, opened a new log
I20260924 07:08:28.758380 17616 raft_consensus.cc:348] T 00000000000000000000000000000000 P a68e296c1fdb43b2863b077ed098de0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68e296c1fdb43b2863b077ed098de0c" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 38265 } }
I20260924 07:08:28.758667 17616 raft_consensus.cc:374] T 00000000000000000000000000000000 P a68e296c1fdb43b2863b077ed098de0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:08:28.758713 17616 raft_consensus.cc:729] T 00000000000000000000000000000000 P a68e296c1fdb43b2863b077ed098de0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a68e296c1fdb43b2863b077ed098de0c, State: Initialized, Role: FOLLOWER
I20260924 07:08:28.759167 17616 consensus_queue.cc:260] T 00000000000000000000000000000000 P a68e296c1fdb43b2863b077ed098de0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68e296c1fdb43b2863b077ed098de0c" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 38265 } }
I20260924 07:08:28.759295 17616 raft_consensus.cc:388] T 00000000000000000000000000000000 P a68e296c1fdb43b2863b077ed098de0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:08:28.759353 17616 raft_consensus.cc:482] T 00000000000000000000000000000000 P a68e296c1fdb43b2863b077ed098de0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:08:28.759443 17616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a68e296c1fdb43b2863b077ed098de0c [term 0 FOLLOWER]: Advancing to term 1
Sep 24 07:08:28 dist-test-slave-2p7j krb5kdc[17535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:08:28.760490 17616 raft_consensus.cc:504] T 00000000000000000000000000000000 P a68e296c1fdb43b2863b077ed098de0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68e296c1fdb43b2863b077ed098de0c" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 38265 } }
I20260924 07:08:28.760833 17616 leader_election.cc:302] T 00000000000000000000000000000000 P a68e296c1fdb43b2863b077ed098de0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a68e296c1fdb43b2863b077ed098de0c; no voters:
I20260924 07:08:28.761195 17616 leader_election.cc:288] T 00000000000000000000000000000000 P a68e296c1fdb43b2863b077ed098de0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:08:28.761299 17621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a68e296c1fdb43b2863b077ed098de0c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:08:28.761509 17621 raft_consensus.cc:686] T 00000000000000000000000000000000 P a68e296c1fdb43b2863b077ed098de0c [term 1 LEADER]: Becoming Leader. State: Replica: a68e296c1fdb43b2863b077ed098de0c, State: Running, Role: LEADER
I20260924 07:08:28.761834 17621 consensus_queue.cc:237] T 00000000000000000000000000000000 P a68e296c1fdb43b2863b077ed098de0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68e296c1fdb43b2863b077ed098de0c" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 38265 } }
I20260924 07:08:28.762979 17616 sys_catalog.cc:564] T 00000000000000000000000000000000 P a68e296c1fdb43b2863b077ed098de0c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:08:28.763041 17622 sys_catalog.cc:455] T 00000000000000000000000000000000 P a68e296c1fdb43b2863b077ed098de0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a68e296c1fdb43b2863b077ed098de0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68e296c1fdb43b2863b077ed098de0c" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 38265 } } }
I20260924 07:08:28.763433 17622 sys_catalog.cc:458] T 00000000000000000000000000000000 P a68e296c1fdb43b2863b077ed098de0c [sys.catalog]: This master's current role is: LEADER
I20260924 07:08:28.763005 17623 sys_catalog.cc:455] T 00000000000000000000000000000000 P a68e296c1fdb43b2863b077ed098de0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a68e296c1fdb43b2863b077ed098de0c. Latest consensus state: current_term: 1 leader_uuid: "a68e296c1fdb43b2863b077ed098de0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68e296c1fdb43b2863b077ed098de0c" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 38265 } } }
I20260924 07:08:28.763794 17623 sys_catalog.cc:458] T 00000000000000000000000000000000 P a68e296c1fdb43b2863b077ed098de0c [sys.catalog]: This master's current role is: LEADER
I20260924 07:08:28.764165 17625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:08:28.766436 17619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:28.750585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34000 (local address 127.3.217.62:38265)
0924 07:08:28.751111 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:28.751129 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:28.751176 (+ 47us) server_negotiation.cc:407] Connection header received
0924 07:08:28.752240 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:28.752266 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:28.752673 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:28.753047 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:28.754261 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:28.755329 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:28.756510 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:28.757011 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:28.757150 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:28.760138 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:28.760144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:28.760157 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:28.760185 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:28.762401 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:28.762976 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:28.762983 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:28.762989 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:28.763281 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:28.763654 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:28.763658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:28.763660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:28.764207 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:28.764393 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:28.765200 (+ 807us) server_negotiation.cc:299] Negotiation successful
0924 07:08:28.765518 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":141,"threads_started":1}
I20260924 07:08:28.768765 17625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:08:28.769778 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:08:28.775995 17625 catalog_manager.cc:1347] Generated new cluster ID: 45d677ce98e94d45ae8a0fac886e1a31
I20260924 07:08:28.776114 17625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:08:28.795647 17625 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:08:28.796753 17625 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:08:28.808243 17625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a68e296c1fdb43b2863b077ed098de0c: Generated new TSK 0
I20260924 07:08:28.809427 17625 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:08:28.827286 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 17548
2026-09-24T07:08:28Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1399 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:08:28 dist-test-slave-2p7j krb5kdc[17643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:08:28 dist-test-slave-2p7j krb5kdc[17643](info): set up 2 sockets
Sep 24 07:08:28 dist-test-slave-2p7j krb5kdc[17643](info): commencing operation
krb5kdc: starting...
W20260924 07:08:29.792505 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:08:29 dist-test-slave-2p7j krb5kdc[17643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:08:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:08:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:08:29.931727 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:42227
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:42227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:08:30.049378 17656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:30.049641 17656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:30.049680 17656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:30.053689 17656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:08:30.053750 17656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:30.053767 17656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:30.053866 17656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:08:30.053934 17656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:08:30.060312 17656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:42227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:42227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:30.061441 17656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:30.062431 17656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:30.071475 17664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:30.071632 17661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:30.071601 17662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:30.072028 17656 server_base.cc:1034] running on GCE node
I20260924 07:08:30.072580 17656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:30.073659 17656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:30.074849 17656 hybrid_clock.cc:648] HybridClock initialized: now 1790233710074822 us; error 49 us; skew 500 ppm
Sep 24 07:08:30 dist-test-slave-2p7j krb5kdc[17643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233710, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:30.078161 17656 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:08:30.079393 17656 webserver.cc:460] Webserver started at http://127.3.217.62:33551/ using document root <none> and password file <none>
I20260924 07:08:30.080327 17656 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:30.080436 17656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:08:30.080654 17656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:08:30.083307 17656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "08e2efa8c00d4b56a6006983b78126fc"
format_stamp: "Formatted at 2026-09-24 07:08:30 on dist-test-slave-2p7j"
server_key: "578f46ee9985c7a954001c6c61be9afe"
server_key_iv: "8e26479df6be03e65316169fb239dc05"
server_key_version: "encryptionkey@0"
I20260924 07:08:30.084261 17656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "08e2efa8c00d4b56a6006983b78126fc"
format_stamp: "Formatted at 2026-09-24 07:08:30 on dist-test-slave-2p7j"
server_key: "578f46ee9985c7a954001c6c61be9afe"
server_key_iv: "8e26479df6be03e65316169fb239dc05"
server_key_version: "encryptionkey@0"
I20260924 07:08:30.088722 17656 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 07:08:30.092301 17671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:30.093849 17656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:08:30.094175 17656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "08e2efa8c00d4b56a6006983b78126fc"
format_stamp: "Formatted at 2026-09-24 07:08:30 on dist-test-slave-2p7j"
server_key: "578f46ee9985c7a954001c6c61be9afe"
server_key_iv: "8e26479df6be03e65316169fb239dc05"
server_key_version: "encryptionkey@0"
I20260924 07:08:30.094323 17656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:30.135456 17656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:30.138912 17656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:30.139065 17656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:30.145857 17656 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:42227
I20260924 07:08:30.146097 17723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:42227 every 8 connection(s)
I20260924 07:08:30.146759 17656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:08:30.149652 17724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:08:30.150017 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 17656
I20260924 07:08:30.150116 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:08:30.150362 3940 external_mini_cluster.cc:1428] Setting key 7da56cc4b3afed837e2a36464b94b0d4
I20260924 07:08:30.156423 17724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc: Bootstrap starting.
I20260924 07:08:30.158746 17724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc: Neither blocks nor log segments found. Creating new log.
Sep 24 07:08:30 dist-test-slave-2p7j krb5kdc[17643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:08:30.159605 17724 log.cc:824] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc: Log is configured to *not* fsync() on all Append() calls
I20260924 07:08:30.161753 17724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc: No bootstrap required, opened a new log
I20260924 07:08:30.165534 17727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:30.151743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38828 (local address 127.3.217.62:42227)
0924 07:08:30.152140 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:30.152158 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:30.152196 (+ 38us) server_negotiation.cc:407] Connection header received
0924 07:08:30.152892 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:30.152922 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:30.153327 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:30.153677 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:30.154319 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.155760 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.156800 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.157448 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.157608 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:30.160104 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:30.160110 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:30.160126 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:30.160154 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:30.161943 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:30.162541 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:30.162550 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:30.162557 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:30.162825 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:30.163247 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:30.163251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:30.163252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:30.163824 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:30.164002 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:30.164391 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 07:08:30.164688 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":97,"threads_started":1}
I20260924 07:08:30.165787 17724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e2efa8c00d4b56a6006983b78126fc" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42227 } }
I20260924 07:08:30.166011 17724 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:08:30.166064 17724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08e2efa8c00d4b56a6006983b78126fc, State: Initialized, Role: FOLLOWER
I20260924 07:08:30.166904 17724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e2efa8c00d4b56a6006983b78126fc" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42227 } }
I20260924 07:08:30.167063 17724 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:08:30.167130 17724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:08:30.167232 17724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:08:30.168177 17724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e2efa8c00d4b56a6006983b78126fc" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42227 } }
I20260924 07:08:30.168481 17724 leader_election.cc:302] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08e2efa8c00d4b56a6006983b78126fc; no voters:
I20260924 07:08:30.168756 17724 leader_election.cc:288] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:08:30.168977 17729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:08:30.169229 17729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 1 LEADER]: Becoming Leader. State: Replica: 08e2efa8c00d4b56a6006983b78126fc, State: Running, Role: LEADER
I20260924 07:08:30.169548 17729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e2efa8c00d4b56a6006983b78126fc" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42227 } }
I20260924 07:08:30.170145 17724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:08:30.171339 17731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08e2efa8c00d4b56a6006983b78126fc. Latest consensus state: current_term: 1 leader_uuid: "08e2efa8c00d4b56a6006983b78126fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e2efa8c00d4b56a6006983b78126fc" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42227 } } }
I20260924 07:08:30.171466 17731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [sys.catalog]: This master's current role is: LEADER
I20260924 07:08:30.171672 17730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08e2efa8c00d4b56a6006983b78126fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e2efa8c00d4b56a6006983b78126fc" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42227 } } }
I20260924 07:08:30.171756 17730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [sys.catalog]: This master's current role is: LEADER
W20260924 07:08:30.175565 17744 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:08:30.175633 17744 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:08:30.175730 17738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:08:30.176726 17738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:08:30.183420 17738 catalog_manager.cc:1347] Generated new cluster ID: 9f4bbd81a8e64ce89928c982f0c03895
I20260924 07:08:30.183576 17738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:08:30.195182 17738 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:08:30.196314 17738 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:08:30.214634 17738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc: Generated new TSK 0
I20260924 07:08:30.215890 17738 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
I20260924 07:08:30.251892 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42227
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:08:30.368660 17750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:30.368906 17750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:30.368968 17750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:30.373346 17750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:30.373415 17750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:30.373495 17750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:08:30.379020 17750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:30.380084 17750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:30.381021 17750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:30.390558 17758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:30.390499 17755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:30.390513 17756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:30.390834 17750 server_base.cc:1034] running on GCE node
I20260924 07:08:30.391306 17750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:30.391991 17750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:30.393162 17750 hybrid_clock.cc:648] HybridClock initialized: now 1790233710393098 us; error 75 us; skew 500 ppm
Sep 24 07:08:30 dist-test-slave-2p7j krb5kdc[17643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233710, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:30.396932 17750 init.cc:377] Logged in from keytab as kudu/127.3.217.1@KRBTEST.COM (short username kudu)
I20260924 07:08:30.398205 17750 webserver.cc:460] Webserver started at http://127.3.217.1:33555/ using document root <none> and password file <none>
I20260924 07:08:30.399003 17750 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:30.399080 17750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:08:30.399252 17750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:08:30.401350 17750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "c915cd7cabc242f9884a2dfc5c20106c"
format_stamp: "Formatted at 2026-09-24 07:08:30 on dist-test-slave-2p7j"
server_key: "b14d3b7563b7989af5e2c0fdaf283802"
server_key_iv: "cc15ffdfe5c66ceb9fe7f060ffb2d594"
server_key_version: "encryptionkey@0"
I20260924 07:08:30.402002 17750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "c915cd7cabc242f9884a2dfc5c20106c"
format_stamp: "Formatted at 2026-09-24 07:08:30 on dist-test-slave-2p7j"
server_key: "b14d3b7563b7989af5e2c0fdaf283802"
server_key_iv: "cc15ffdfe5c66ceb9fe7f060ffb2d594"
server_key_version: "encryptionkey@0"
I20260924 07:08:30.405767 17750 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:08:30.408597 17765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:30.410190 17750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:08:30.410307 17750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "c915cd7cabc242f9884a2dfc5c20106c"
format_stamp: "Formatted at 2026-09-24 07:08:30 on dist-test-slave-2p7j"
server_key: "b14d3b7563b7989af5e2c0fdaf283802"
server_key_iv: "cc15ffdfe5c66ceb9fe7f060ffb2d594"
server_key_version: "encryptionkey@0"
I20260924 07:08:30.410414 17750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:30.447153 17750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:30.450661 17750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:30.450830 17750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:30.451407 17750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:08:30.452306 17750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:08:30.452369 17750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:30.452414 17750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:08:30.452446 17750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:30.462723 17750 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:38149
I20260924 07:08:30.462740 17878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:38149 every 8 connection(s)
I20260924 07:08:30.463662 17750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:08:30.470207 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 17750
I20260924 07:08:30.470350 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:08:30.470794 3940 external_mini_cluster.cc:1428] Setting key 9b67115f499db2b0dfc8ead785021228
Sep 24 07:08:30 dist-test-slave-2p7j krb5kdc[17643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233710, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:08:30.479229 17727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:30.465574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:41255 (local address 127.3.217.62:42227)
0924 07:08:30.465709 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:30.465714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:30.466709 (+ 995us) server_negotiation.cc:407] Connection header received
0924 07:08:30.467552 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:30.467555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:30.467600 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:30.467672 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:30.469440 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.470187 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.472174 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.472460 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.472563 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:30.475430 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:30.475433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:30.475435 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:30.475459 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:30.476792 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:30.477436 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:30.477439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:30.477440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:30.477519 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:30.477964 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:30.477969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:30.477971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:30.478218 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:30.478304 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:30.478978 (+ 674us) server_negotiation.cc:299] Negotiation successful
0924 07:08:30.479100 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:08:30.480607 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:30.465949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.1:41255)
0924 07:08:30.466576 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:30.466616 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:30.467349 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:30.467810 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:30.467817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:30.468539 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:30.469220 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:30.469241 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.470491 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.470496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:30.472011 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:30.472027 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.472660 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.472667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:30.472969 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:30.473579 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:30.473596 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:30.475264 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:30.476915 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:30.476922 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:30.476937 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:30.477291 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:30.477624 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:30.477627 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:30.477630 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:30.477821 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:30.478328 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:30.478336 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:30.478688 (+ 352us) client_negotiation.cc:769] Sending connection context
0924 07:08:30.478922 (+ 234us) client_negotiation.cc:240] Negotiation successful
0924 07:08:30.479447 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.3.217.2@KRBTEST.COM; defaulting to no policy
I20260924 07:08:30.481891 17879 heartbeater.cc:347] Connected to a master server at 127.3.217.62:42227
I20260924 07:08:30.482131 17879 heartbeater.cc:464] Registering TS with master...
I20260924 07:08:30.482654 17879 heartbeater.cc:511] Master 127.3.217.62:42227 requested a full tablet report, sending...
I20260924 07:08:30.484509 17688 ts_manager.cc:194] Registered new tserver with Master: c915cd7cabc242f9884a2dfc5c20106c (127.3.217.1:38149)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.2@KRBTEST.COM" created.
I20260924 07:08:30.485668 17688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.1@KRBTEST.COM'} at 127.3.217.1:41255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
I20260924 07:08:30.503882 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42227
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:08:30.617064 17884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:30.617311 17884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:30.617368 17884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:30.621290 17884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:30.621347 17884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:30.621426 17884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:08:30.627990 17884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:30.629513 17884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:30.630795 17884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:30.641476 17890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:30.641582 17884 server_base.cc:1034] running on GCE node
W20260924 07:08:30.641762 17892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:30.641487 17889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:30.642187 17884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:30.642762 17884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:30.643930 17884 hybrid_clock.cc:648] HybridClock initialized: now 1790233710643911 us; error 35 us; skew 500 ppm
Sep 24 07:08:30 dist-test-slave-2p7j krb5kdc[17643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233710, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:30.653123 17884 init.cc:377] Logged in from keytab as kudu/127.3.217.2@KRBTEST.COM (short username kudu)
I20260924 07:08:30.654120 17884 webserver.cc:460] Webserver started at http://127.3.217.2:45321/ using document root <none> and password file <none>
I20260924 07:08:30.654682 17884 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:30.654755 17884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:08:30.654973 17884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:08:30.657035 17884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "55769f9844444e4a9ce9d7062d8efcf4"
format_stamp: "Formatted at 2026-09-24 07:08:30 on dist-test-slave-2p7j"
server_key: "de1b90e3e363e91ee6b76fa8752cf7c2"
server_key_iv: "992c96dbd200865d84f9e8f65af179ea"
server_key_version: "encryptionkey@0"
I20260924 07:08:30.657577 17884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "55769f9844444e4a9ce9d7062d8efcf4"
format_stamp: "Formatted at 2026-09-24 07:08:30 on dist-test-slave-2p7j"
server_key: "de1b90e3e363e91ee6b76fa8752cf7c2"
server_key_iv: "992c96dbd200865d84f9e8f65af179ea"
server_key_version: "encryptionkey@0"
I20260924 07:08:30.661332 17884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:08:30.664047 17899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:30.665460 17884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:08:30.665578 17884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "55769f9844444e4a9ce9d7062d8efcf4"
format_stamp: "Formatted at 2026-09-24 07:08:30 on dist-test-slave-2p7j"
server_key: "de1b90e3e363e91ee6b76fa8752cf7c2"
server_key_iv: "992c96dbd200865d84f9e8f65af179ea"
server_key_version: "encryptionkey@0"
I20260924 07:08:30.665669 17884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:30.682952 17884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:30.685884 17884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:30.686053 17884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:30.686614 17884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:08:30.687407 17884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:08:30.687453 17884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:30.687534 17884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:08:30.687561 17884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:30.697006 17884 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:35231
I20260924 07:08:30.697052 18012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:35231 every 8 connection(s)
I20260924 07:08:30.698067 17884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:08:30.701941 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 17884
I20260924 07:08:30.702040 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:08:30.702363 3940 external_mini_cluster.cc:1428] Setting key f431bac9c949c334cc9d45825f06dde8
Sep 24 07:08:30 dist-test-slave-2p7j krb5kdc[17643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233710, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.217.3@KRBTEST.COM; defaulting to no policy
I20260924 07:08:30.713984 17727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:30.700104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:43895 (local address 127.3.217.62:42227)
0924 07:08:30.700254 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:30.700258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:30.701242 (+ 984us) server_negotiation.cc:407] Connection header received
0924 07:08:30.702248 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:30.702251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:30.702302 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:30.702395 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:30.703730 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.704529 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.706119 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.706372 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.706466 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:30.709607 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:30.709611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:30.709615 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:30.709666 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:30.711398 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:30.712107 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:30.712110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:30.712112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:30.712254 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:30.712624 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:30.712627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:30.712628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:30.712850 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:30.712939 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:30.713633 (+ 694us) server_negotiation.cc:299] Negotiation successful
0924 07:08:30.713789 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 07:08:30.715250 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:30.700461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.2:43895)
0924 07:08:30.701100 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:30.701138 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:30.702018 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:30.702592 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:30.702600 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:30.702979 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:30.703544 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:30.703560 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.704699 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.704702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:30.705986 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:30.705998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.706545 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.706549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:30.706690 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:30.707471 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:30.707487 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:30.709322 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:30.711598 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:30.711606 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:30.711625 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:30.711992 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:30.712351 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:30.712354 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:30.712357 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:30.712511 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:30.712954 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:30.712962 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:30.713321 (+ 359us) client_negotiation.cc:769] Sending connection context
0924 07:08:30.713576 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 07:08:30.714032 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":118,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.3@KRBTEST.COM" created.
I20260924 07:08:30.716740 18013 heartbeater.cc:347] Connected to a master server at 127.3.217.62:42227
I20260924 07:08:30.716977 18013 heartbeater.cc:464] Registering TS with master...
I20260924 07:08:30.717458 18013 heartbeater.cc:511] Master 127.3.217.62:42227 requested a full tablet report, sending...
I20260924 07:08:30.718457 17688 ts_manager.cc:194] Registered new tserver with Master: 55769f9844444e4a9ce9d7062d8efcf4 (127.3.217.2:35231)
I20260924 07:08:30.719156 17688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.2@KRBTEST.COM'} at 127.3.217.2:43895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
I20260924 07:08:30.735391 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42227
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:08:30.857679 18018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:30.858031 18018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:30.858081 18018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:30.863471 18018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:30.863584 18018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:30.863663 18018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:08:30.869844 18018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:30.870931 18018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:30.871996 18018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:30.882303 18023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:30.882369 18026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:30.882369 18024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:30.882629 18018 server_base.cc:1034] running on GCE node
I20260924 07:08:30.883236 18018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:30.883932 18018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:30.885116 18018 hybrid_clock.cc:648] HybridClock initialized: now 1790233710885103 us; error 39 us; skew 500 ppm
Sep 24 07:08:30 dist-test-slave-2p7j krb5kdc[17643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233710, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:30.887984 18018 init.cc:377] Logged in from keytab as kudu/127.3.217.3@KRBTEST.COM (short username kudu)
I20260924 07:08:30.888850 18018 webserver.cc:460] Webserver started at http://127.3.217.3:41947/ using document root <none> and password file <none>
I20260924 07:08:30.889396 18018 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:30.889463 18018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:08:30.889644 18018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:08:30.892022 18018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "8c837b29bb524c38890cc1328b014f4b"
format_stamp: "Formatted at 2026-09-24 07:08:30 on dist-test-slave-2p7j"
server_key: "ea45bdf74a71496307e61bf091cc8677"
server_key_iv: "b999b8530ece4a1072438bd5472966f7"
server_key_version: "encryptionkey@0"
I20260924 07:08:30.892901 18018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "8c837b29bb524c38890cc1328b014f4b"
format_stamp: "Formatted at 2026-09-24 07:08:30 on dist-test-slave-2p7j"
server_key: "ea45bdf74a71496307e61bf091cc8677"
server_key_iv: "b999b8530ece4a1072438bd5472966f7"
server_key_version: "encryptionkey@0"
I20260924 07:08:30.898027 18018 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 07:08:30.901494 18033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:30.903266 18018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:08:30.903453 18018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "8c837b29bb524c38890cc1328b014f4b"
format_stamp: "Formatted at 2026-09-24 07:08:30 on dist-test-slave-2p7j"
server_key: "ea45bdf74a71496307e61bf091cc8677"
server_key_iv: "b999b8530ece4a1072438bd5472966f7"
server_key_version: "encryptionkey@0"
I20260924 07:08:30.903661 18018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:30.923652 18018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:30.927851 18018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:30.928066 18018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:30.928660 18018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:08:30.929495 18018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:08:30.929543 18018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:30.929596 18018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:08:30.929626 18018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:30.938431 18018 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:35729
I20260924 07:08:30.938486 18146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:35729 every 8 connection(s)
I20260924 07:08:30.939285 18018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:08:30.944075 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 18018
I20260924 07:08:30.944219 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:08:30.944528 3940 external_mini_cluster.cc:1428] Setting key c06f97dd605b63492dcc31dabbe6ac5d
Sep 24 07:08:30 dist-test-slave-2p7j krb5kdc[17643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233710, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:08:30.953398 17727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:30.941069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:46577 (local address 127.3.217.62:42227)
0924 07:08:30.941207 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:30.941211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:30.942169 (+ 958us) server_negotiation.cc:407] Connection header received
0924 07:08:30.943128 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:30.943132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:30.943186 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:30.943261 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:30.944603 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.945263 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.946507 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.946773 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.946854 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:30.949397 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:30.949399 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:30.949402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:30.949422 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:30.950623 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:30.951087 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:30.951090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:30.951091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:30.951170 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:30.951887 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:30.951889 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:30.951891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:30.952091 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:30.952183 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:30.953045 (+ 862us) server_negotiation.cc:299] Negotiation successful
0924 07:08:30.953232 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:08:30.954452 18149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:30.941366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.3:46577)
0924 07:08:30.942017 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:30.942055 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:30.942902 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:30.943422 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:30.943429 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:30.943816 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:30.944443 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:30.944460 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.945377 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.945382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:30.946390 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:30.946403 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.946921 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.946925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:30.947062 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:30.947678 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:30.947694 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:30.949232 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:30.950726 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:30.950730 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:30.950740 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:30.950989 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:30.951427 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:30.951432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:30.951434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:30.951747 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:30.952251 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:30.952301 (+ 50us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:30.952663 (+ 362us) client_negotiation.cc:769] Sending connection context
0924 07:08:30.952900 (+ 237us) client_negotiation.cc:240] Negotiation successful
0924 07:08:30.953251 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":120,"threads_started":1}
I20260924 07:08:30.955821 18147 heartbeater.cc:347] Connected to a master server at 127.3.217.62:42227
I20260924 07:08:30.956046 18147 heartbeater.cc:464] Registering TS with master...
I20260924 07:08:30.956674 18147 heartbeater.cc:511] Master 127.3.217.62:42227 requested a full tablet report, sending...
I20260924 07:08:30.957948 17688 ts_manager.cc:194] Registered new tserver with Master: 8c837b29bb524c38890cc1328b014f4b (127.3.217.3:35729)
I20260924 07:08:30.958678 17688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.3@KRBTEST.COM'} at 127.3.217.3:46577
I20260924 07:08:30.958809 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:08:30.969775 17727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:30.961721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38838 (local address 127.3.217.62:42227)
0924 07:08:30.961910 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:30.961914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:30.962185 (+ 271us) server_negotiation.cc:407] Connection header received
0924 07:08:30.962355 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:30.962359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:30.962423 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:30.962523 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:30.963250 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.963901 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.965146 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.965505 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.965620 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:30.966958 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:30.966962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:30.966965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:30.966987 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:30.968287 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:30.968642 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:30.968645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:30.968647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:30.968719 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:30.968972 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:30.968975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:30.968977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:30.969183 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:30.969298 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:30.969505 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 07:08:30.969619 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 07:08:30.973606 17688 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:08:30.976554 17688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:08:30.993016 18158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:30.986607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:38149 (local address 127.0.0.1:51832)
0924 07:08:30.987781 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:30.987793 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:30.987853 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:30.988647 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:30.988652 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:30.988687 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:08:30.988848 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:30.988855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.989974 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.989977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:30.991372 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:30.991385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.992446 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.992450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:30.992562 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:30.992607 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 07:08:30.992725 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 07:08:30.992859 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1123,"mutex_wait_us":961,"thread_start_us":151,"threads_started":1}
I20260924 07:08:30.993084 18160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:30.986918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51832 (local address 127.3.217.1:38149)
0924 07:08:30.988054 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:30.988062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:30.988090 (+ 28us) server_negotiation.cc:407] Connection header received
0924 07:08:30.988186 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:30.988192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:30.988463 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:30.988662 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:08:30.988974 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.989852 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.991536 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.992313 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.992438 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:30.992626 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:30.992765 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 07:08:30.992893 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1011,"thread_start_us":91,"threads_started":1}
I20260924 07:08:30.993523 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:30.986711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:35231 (local address 127.0.0.1:43758)
0924 07:08:30.987733 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:30.987747 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:30.987830 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:30.988928 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:30.988932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:30.988957 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:08:30.989079 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:30.989086 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.990517 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.990520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:30.991980 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:30.991987 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.993067 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.993070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:30.993168 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:30.993197 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 07:08:30.993262 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 07:08:30.993334 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":20,"thread_start_us":105,"threads_started":1}
I20260924 07:08:30.993778 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:30.987396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43758 (local address 127.3.217.2:35231)
0924 07:08:30.988330 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:30.988341 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:30.988362 (+ 21us) server_negotiation.cc:407] Connection header received
0924 07:08:30.988437 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:30.988442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:30.988610 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:30.988743 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:08:30.989228 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.990376 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.992078 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.992776 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.992866 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:30.993004 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:30.993410 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 07:08:30.993513 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":148,"threads_started":1}
I20260924 07:08:30.993346 18159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:30.986607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:35729 (local address 127.0.0.1:50372)
0924 07:08:30.987070 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:30.987116 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:30.987236 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:30.988777 (+ 1541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:30.988780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:30.988794 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:08:30.988868 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:30.988874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.990459 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.990463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:30.991795 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:30.991807 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.992929 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.992933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:30.993033 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:30.993060 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 07:08:30.993110 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 07:08:30.993202 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":43,"thread_start_us":136,"threads_started":1}
I20260924 07:08:30.994002 18163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:30.987597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50372 (local address 127.3.217.3:35729)
0924 07:08:30.988328 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:30.988341 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:30.988366 (+ 25us) server_negotiation.cc:407] Connection header received
0924 07:08:30.988438 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:30.988443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:30.988634 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:30.989003 (+ 369us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:08:30.989124 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.990282 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.991955 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:30.992785 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:30.993468 (+ 683us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:30.993618 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:30.993669 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 07:08:30.993828 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":114,"threads_started":1}
I20260924 07:08:30.996492 17813 tablet_service.cc:1467] Processing CreateTablet for tablet 76f38037b25d41c78c0f41555d27b0ee (DEFAULT_TABLE table=test-table [id=c27f1f55be904c80be17fe81c8ff6a19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:08:30.996556 18078 tablet_service.cc:1467] Processing CreateTablet for tablet 76f38037b25d41c78c0f41555d27b0ee (DEFAULT_TABLE table=test-table [id=c27f1f55be904c80be17fe81c8ff6a19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:08:30.996557 17946 tablet_service.cc:1467] Processing CreateTablet for tablet 76f38037b25d41c78c0f41555d27b0ee (DEFAULT_TABLE table=test-table [id=c27f1f55be904c80be17fe81c8ff6a19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:08:30.997467 17813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f38037b25d41c78c0f41555d27b0ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:08:30.997540 17946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f38037b25d41c78c0f41555d27b0ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:08:30.997540 18078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f38037b25d41c78c0f41555d27b0ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:08:31.004387 18164 tablet_bootstrap.cc:492] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c: Bootstrap starting.
I20260924 07:08:31.005570 18165 tablet_bootstrap.cc:492] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b: Bootstrap starting.
I20260924 07:08:31.006641 18164 tablet_bootstrap.cc:654] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c: Neither blocks nor log segments found. Creating new log.
I20260924 07:08:31.007251 18165 tablet_bootstrap.cc:654] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b: Neither blocks nor log segments found. Creating new log.
I20260924 07:08:31.007411 18164 log.cc:824] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:08:31.007902 18166 tablet_bootstrap.cc:492] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4: Bootstrap starting.
I20260924 07:08:31.008476 18165 log.cc:824] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b: Log is configured to *not* fsync() on all Append() calls
I20260924 07:08:31.009994 18164 tablet_bootstrap.cc:492] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c: No bootstrap required, opened a new log
I20260924 07:08:31.010196 18164 ts_tablet_manager.cc:1397] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:08:31.010558 18166 tablet_bootstrap.cc:654] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4: Neither blocks nor log segments found. Creating new log.
I20260924 07:08:31.010689 18165 tablet_bootstrap.cc:492] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b: No bootstrap required, opened a new log
I20260924 07:08:31.010869 18165 ts_tablet_manager.cc:1397] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260924 07:08:31.011418 18166 log.cc:824] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:08:31.014637 18166 tablet_bootstrap.cc:492] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4: No bootstrap required, opened a new log
I20260924 07:08:31.014954 18166 ts_tablet_manager.cc:1397] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 07:08:31.014825 18165 raft_consensus.cc:348] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:31.015079 18165 raft_consensus.cc:374] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:08:31.015132 18165 raft_consensus.cc:729] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c837b29bb524c38890cc1328b014f4b, State: Initialized, Role: FOLLOWER
I20260924 07:08:31.015582 18165 consensus_queue.cc:260] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:31.016010 18164 raft_consensus.cc:348] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:31.016260 18164 raft_consensus.cc:374] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:08:31.016307 18164 raft_consensus.cc:729] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c915cd7cabc242f9884a2dfc5c20106c, State: Initialized, Role: FOLLOWER
I20260924 07:08:31.016542 18165 ts_tablet_manager.cc:1428] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:08:31.016738 18164 consensus_queue.cc:260] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:31.016872 18147 heartbeater.cc:503] Master 127.3.217.62:42227 was elected leader, sending a full tablet report...
I20260924 07:08:31.017382 17879 heartbeater.cc:503] Master 127.3.217.62:42227 was elected leader, sending a full tablet report...
I20260924 07:08:31.017671 18164 ts_tablet_manager.cc:1428] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260924 07:08:31.018945 18166 raft_consensus.cc:348] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:31.019173 18166 raft_consensus.cc:374] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:08:31.019227 18166 raft_consensus.cc:729] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55769f9844444e4a9ce9d7062d8efcf4, State: Initialized, Role: FOLLOWER
I20260924 07:08:31.019595 18166 consensus_queue.cc:260] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:31.020157 18013 heartbeater.cc:503] Master 127.3.217.62:42227 was elected leader, sending a full tablet report...
I20260924 07:08:31.021227 18166 ts_tablet_manager.cc:1428] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260924 07:08:31.055855 18170 raft_consensus.cc:482] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:08:31.056052 18170 raft_consensus.cc:504] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:31.057292 18170 leader_election.cc:288] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c915cd7cabc242f9884a2dfc5c20106c (127.3.217.1:38149), 55769f9844444e4a9ce9d7062d8efcf4 (127.3.217.2:35231)
I20260924 07:08:31.062588 18149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:31.057513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:38149 (local address 127.3.217.3:48041)
0924 07:08:31.057677 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:31.057692 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:31.057776 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:31.058059 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:31.058062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:31.058076 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:08:31.058188 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:31.058193 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.059227 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.059231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:31.060716 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:31.060727 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.061731 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.061738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:31.062014 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:31.062080 (+ 66us) client_negotiation.cc:769] Sending connection context
0924 07:08:31.062224 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 07:08:31.062364 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":257}
I20260924 07:08:31.062615 18160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:31.057637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:48041 (local address 127.3.217.1:38149)
0924 07:08:31.057752 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:31.057756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:31.057770 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:08:31.057901 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:31.057904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:31.057947 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:31.058017 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:08:31.058285 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.059091 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.060867 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.061524 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.061614 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:31.061678 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:31.062334 (+ 656us) server_negotiation.cc:299] Negotiation successful
0924 07:08:31.062462 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:08:31.062615 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:31.057660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:42201 (local address 127.3.217.2:35231)
0924 07:08:31.057783 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:31.057787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:31.058036 (+ 249us) server_negotiation.cc:407] Connection header received
0924 07:08:31.058120 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:31.058124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:31.058162 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:31.058217 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:08:31.058428 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.059144 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.060867 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.061485 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.061614 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:31.061678 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:31.062334 (+ 656us) server_negotiation.cc:299] Negotiation successful
0924 07:08:31.062462 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:08:31.062602 18173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:31.057560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:35231 (local address 127.3.217.3:42201)
0924 07:08:31.057947 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:31.057957 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:31.058040 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:31.058247 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:31.058250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:31.058265 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:08:31.058347 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:31.058351 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.059244 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.059247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:31.060716 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:31.060727 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.061731 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.061740 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:31.062017 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:31.062080 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 07:08:31.062224 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 07:08:31.062364 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":155,"thread_start_us":138,"threads_started":1}
I20260924 07:08:31.063450 17833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f38037b25d41c78c0f41555d27b0ee" candidate_uuid: "8c837b29bb524c38890cc1328b014f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c915cd7cabc242f9884a2dfc5c20106c" is_pre_election: true
I20260924 07:08:31.063638 17967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f38037b25d41c78c0f41555d27b0ee" candidate_uuid: "8c837b29bb524c38890cc1328b014f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55769f9844444e4a9ce9d7062d8efcf4" is_pre_election: true
I20260924 07:08:31.063823 17833 raft_consensus.cc:2454] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c837b29bb524c38890cc1328b014f4b in term 0.
I20260924 07:08:31.064023 17967 raft_consensus.cc:2454] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c837b29bb524c38890cc1328b014f4b in term 0.
I20260924 07:08:31.064361 18035 leader_election.cc:302] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c837b29bb524c38890cc1328b014f4b, c915cd7cabc242f9884a2dfc5c20106c; no voters:
I20260924 07:08:31.064638 18170 raft_consensus.cc:2781] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:08:31.064720 18170 raft_consensus.cc:482] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:08:31.064767 18170 raft_consensus.cc:3037] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:08:31.065826 18170 raft_consensus.cc:504] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:31.066148 18170 leader_election.cc:288] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b [CANDIDATE]: Term 1 election: Requested vote from peers c915cd7cabc242f9884a2dfc5c20106c (127.3.217.1:38149), 55769f9844444e4a9ce9d7062d8efcf4 (127.3.217.2:35231)
I20260924 07:08:31.066722 17967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f38037b25d41c78c0f41555d27b0ee" candidate_uuid: "8c837b29bb524c38890cc1328b014f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55769f9844444e4a9ce9d7062d8efcf4"
I20260924 07:08:31.066656 17833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f38037b25d41c78c0f41555d27b0ee" candidate_uuid: "8c837b29bb524c38890cc1328b014f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c915cd7cabc242f9884a2dfc5c20106c"
I20260924 07:08:31.066854 17967 raft_consensus.cc:3037] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:08:31.066854 17833 raft_consensus.cc:3037] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:08:31.068082 17967 raft_consensus.cc:2454] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c837b29bb524c38890cc1328b014f4b in term 1.
I20260924 07:08:31.068081 17833 raft_consensus.cc:2454] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c837b29bb524c38890cc1328b014f4b in term 1.
I20260924 07:08:31.068542 18035 leader_election.cc:302] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c837b29bb524c38890cc1328b014f4b, c915cd7cabc242f9884a2dfc5c20106c; no voters:
I20260924 07:08:31.068820 18170 raft_consensus.cc:2781] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:08:31.069135 18170 raft_consensus.cc:686] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b [term 1 LEADER]: Becoming Leader. State: Replica: 8c837b29bb524c38890cc1328b014f4b, State: Running, Role: LEADER
I20260924 07:08:31.069717 18170 consensus_queue.cc:237] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:31.073356 17688 catalog_manager.cc:5495] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c837b29bb524c38890cc1328b014f4b (127.3.217.3). New cstate: current_term: 1 leader_uuid: "8c837b29bb524c38890cc1328b014f4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } health_report { overall_health: UNKNOWN } } }
I20260924 07:08:31.093356 18163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:31.089365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50378 (local address 127.3.217.3:35729)
0924 07:08:31.089498 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:31.089502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:31.089517 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:08:31.089580 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:31.089582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:31.089635 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:31.089725 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:08:31.090183 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.090675 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.091579 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.091784 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.091862 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:31.092239 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:08:31.092648 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:08:31.092740 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:31.092996 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 07:08:31.093118 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 07:08:31.100440 17833 raft_consensus.cc:1261] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Refusing update from remote peer 8c837b29bb524c38890cc1328b014f4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:08:31.100924 17967 raft_consensus.cc:1261] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4 [term 1 FOLLOWER]: Refusing update from remote peer 8c837b29bb524c38890cc1328b014f4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:08:31.101114 18170 consensus_queue.cc:1035] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:08:31.101686 18170 consensus_queue.cc:1035] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:08:31.110428 18179 mvcc.cc:205] Tried to move back new op lower bound from 7332797280659742720 to 7332797280545763328. Current Snapshot: MvccSnapshot[applied={T|T < 7332797280659742720}]
I20260924 07:08:31.117552 17687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:08:31.117905 17687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:08:31.120647 17687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc: Sending DeleteTablet for 3 replicas of tablet 76f38037b25d41c78c0f41555d27b0ee
I20260924 07:08:31.121402 17946 tablet_service.cc:1514] Processing DeleteTablet for tablet 76f38037b25d41c78c0f41555d27b0ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:08:31 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:43758
I20260924 07:08:31.121417 17813 tablet_service.cc:1514] Processing DeleteTablet for tablet 76f38037b25d41c78c0f41555d27b0ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:08:31 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:51832
I20260924 07:08:31.121534 18078 tablet_service.cc:1514] Processing DeleteTablet for tablet 76f38037b25d41c78c0f41555d27b0ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:08:31 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:50372
I20260924 07:08:31.121882 18187 tablet_replica.cc:330] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c: stopping tablet replica
I20260924 07:08:31.121932 18188 tablet_replica.cc:330] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b: stopping tablet replica
I20260924 07:08:31.122094 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 17750
I20260924 07:08:31.122219 18188 raft_consensus.cc:2229] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:08:31.122347 18187 raft_consensus.cc:2229] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:08:31.122604 18188 raft_consensus.cc:2258] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:08:31.122933 18189 tablet_replica.cc:330] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4: stopping tablet replica
I20260924 07:08:31.123178 18189 raft_consensus.cc:2229] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:08:31.123466 18189 raft_consensus.cc:2258] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:08:31.125577 18188 ts_tablet_manager.cc:1905] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:08:31.128401 18188 ts_tablet_manager.cc:1918] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:08:31.128495 18188 log.cc:1196] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/76f38037b25d41c78c0f41555d27b0ee
I20260924 07:08:31.128796 18188 ts_tablet_manager.cc:1939] T 76f38037b25d41c78c0f41555d27b0ee P 8c837b29bb524c38890cc1328b014f4b: Deleting consensus metadata
I20260924 07:08:31.129024 18189 ts_tablet_manager.cc:1905] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 07:08:31.129292 17673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.217.1:38149 (error 108)
I20260924 07:08:31.129868 17675 catalog_manager.cc:4841] TS 8c837b29bb524c38890cc1328b014f4b (127.3.217.3:35729): tablet 76f38037b25d41c78c0f41555d27b0ee (table test-table [id=c27f1f55be904c80be17fe81c8ff6a19]) successfully deleted
I20260924 07:08:31.130307 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 17884
I20260924 07:08:31.130932 18158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:31.130341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:38149 (local address 127.0.0.1:51834)
0924 07:08:31.130788 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:31.130853 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.1:38149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366}
W20260924 07:08:31.131361 17673 catalog_manager.cc:4584] TS c915cd7cabc242f9884a2dfc5c20106c (127.3.217.1:38149): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 76f38037b25d41c78c0f41555d27b0ee: Network error: Client connection negotiation failed: client connection to 127.3.217.1:38149: connect: Connection refused (error 111)
W20260924 07:08:31.136201 18158 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:08:31.135950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:35231 (local address 127.0.0.1:43772)
0924 07:08:31.136059 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:31.136115 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.2:35231: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 07:08:31.136396 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 18018
I20260924 07:08:31.141518 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 17656
I20260924 07:08:31.147145 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:42227
--webserver_interface=127.3.217.62
--webserver_port=33551
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:42227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:08:31.264825 18191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:31.265086 18191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:31.265142 18191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:31.269101 18191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:08:31.269155 18191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:31.269181 18191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:31.269204 18191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:08:31.269232 18191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:08:31.274459 18191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:42227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:42227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:31.275457 18191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:31.276445 18191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:31.289445 18197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:31.289390 18198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:31.289420 18200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:31.289819 18191 server_base.cc:1034] running on GCE node
I20260924 07:08:31.290360 18191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:31.291430 18191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:31.292614 18191 hybrid_clock.cc:648] HybridClock initialized: now 1790233711292592 us; error 38 us; skew 500 ppm
Sep 24 07:08:31 dist-test-slave-2p7j krb5kdc[17643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233711, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:31.295280 18191 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:08:31.296222 18191 webserver.cc:460] Webserver started at http://127.3.217.62:33551/ using document root <none> and password file <none>
I20260924 07:08:31.296761 18191 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:31.296839 18191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:08:31.301321 18191 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 07:08:31.304021 18207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:31.306061 18191 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:08:31.306284 18191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "08e2efa8c00d4b56a6006983b78126fc"
format_stamp: "Formatted at 2026-09-24 07:08:30 on dist-test-slave-2p7j"
server_key: "578f46ee9985c7a954001c6c61be9afe"
server_key_iv: "8e26479df6be03e65316169fb239dc05"
server_key_version: "encryptionkey@0"
I20260924 07:08:31.306964 18191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:31.325441 18191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:31.328825 18191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:31.329010 18191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:31.337452 18191 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:42227
I20260924 07:08:31.337733 18259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:42227 every 8 connection(s)
I20260924 07:08:31.338582 18191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:08:31.343412 18260 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:08:31.345055 18260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc: Bootstrap starting.
I20260924 07:08:31.345371 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 18191
I20260924 07:08:31.345747 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:38149
--local_ip_for_outbound_sockets=127.3.217.1
--tserver_master_addrs=127.3.217.62:42227
--webserver_port=33555
--webserver_interface=127.3.217.1
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:08:31.366173 18260 log.cc:824] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc: Log is configured to *not* fsync() on all Append() calls
I20260924 07:08:31.377395 18260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:08:31.377794 18260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc: Bootstrap complete.
I20260924 07:08:31.383095 18260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e2efa8c00d4b56a6006983b78126fc" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42227 } }
I20260924 07:08:31.383635 18260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08e2efa8c00d4b56a6006983b78126fc, State: Initialized, Role: FOLLOWER
I20260924 07:08:31.384186 18260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e2efa8c00d4b56a6006983b78126fc" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42227 } }
I20260924 07:08:31.384332 18260 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:08:31.384402 18260 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:08:31.384498 18260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:08:31.386926 18260 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e2efa8c00d4b56a6006983b78126fc" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42227 } }
I20260924 07:08:31.387279 18260 leader_election.cc:302] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08e2efa8c00d4b56a6006983b78126fc; no voters:
I20260924 07:08:31.387627 18260 leader_election.cc:288] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 07:08:31.388099 18264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 2 FOLLOWER]: Leader election won for term 2
I20260924 07:08:31.388584 18264 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 2 LEADER]: Becoming Leader. State: Replica: 08e2efa8c00d4b56a6006983b78126fc, State: Running, Role: LEADER
I20260924 07:08:31.388835 18260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:08:31.388991 18264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e2efa8c00d4b56a6006983b78126fc" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42227 } }
I20260924 07:08:31.390403 18265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "08e2efa8c00d4b56a6006983b78126fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e2efa8c00d4b56a6006983b78126fc" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42227 } } }
I20260924 07:08:31.390506 18265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [sys.catalog]: This master's current role is: LEADER
I20260924 07:08:31.390360 18266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08e2efa8c00d4b56a6006983b78126fc. Latest consensus state: current_term: 2 leader_uuid: "08e2efa8c00d4b56a6006983b78126fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e2efa8c00d4b56a6006983b78126fc" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42227 } } }
I20260924 07:08:31.390731 18266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [sys.catalog]: This master's current role is: LEADER
I20260924 07:08:31.391070 18278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:08:31.395895 18278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:08:31.396518 18278 catalog_manager.cc:1259] Loaded cluster ID: 9f4bbd81a8e64ce89928c982f0c03895
I20260924 07:08:31.396591 18278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:08:31.397966 18278 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 07:08:31.402838 18278 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 07:08:31.408613 18278 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:08:31.409335 18278 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc: Loaded TSK: 0
I20260924 07:08:31.410497 18278 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:08:31.472065 18262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:31.472291 18262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:31.472333 18262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:31.476262 18262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:31.476316 18262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:31.476377 18262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:08:31.482622 18262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:38149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=33555
--tserver_master_addrs=127.3.217.62:42227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:31.483578 18262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:31.484558 18262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:31.493847 18288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:31.493873 18290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:31.493873 18287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:31.494194 18262 server_base.cc:1034] running on GCE node
I20260924 07:08:31.494841 18262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:31.495591 18262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:31.496784 18262 hybrid_clock.cc:648] HybridClock initialized: now 1790233711496763 us; error 29 us; skew 500 ppm
Sep 24 07:08:31 dist-test-slave-2p7j krb5kdc[17643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233711, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:31.500066 18262 init.cc:377] Logged in from keytab as kudu/127.3.217.1@KRBTEST.COM (short username kudu)
I20260924 07:08:31.501134 18262 webserver.cc:460] Webserver started at http://127.3.217.1:33555/ using document root <none> and password file <none>
I20260924 07:08:31.501806 18262 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:31.501914 18262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:08:31.506953 18262 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260924 07:08:31.509568 18297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:31.511274 18262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:08:31.511408 18262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "c915cd7cabc242f9884a2dfc5c20106c"
format_stamp: "Formatted at 2026-09-24 07:08:30 on dist-test-slave-2p7j"
server_key: "b14d3b7563b7989af5e2c0fdaf283802"
server_key_iv: "cc15ffdfe5c66ceb9fe7f060ffb2d594"
server_key_version: "encryptionkey@0"
I20260924 07:08:31.512247 18262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:31.544584 18262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:31.547892 18262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:31.548067 18262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:31.548631 18262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:08:31.549676 18304 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 07:08:31.553491 18262 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 07:08:31.553582 18262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260924 07:08:31.553649 18262 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 07:08:31.556111 18262 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 07:08:31.556159 18262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260924 07:08:31.556432 18304 tablet_bootstrap.cc:492] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c: Bootstrap starting.
I20260924 07:08:31.565599 18262 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:38149
I20260924 07:08:31.565913 18411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:38149 every 8 connection(s)
I20260924 07:08:31.566522 18262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:08:31.573395 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 18262
I20260924 07:08:31.573858 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:35231
--local_ip_for_outbound_sockets=127.3.217.2
--tserver_master_addrs=127.3.217.62:42227
--webserver_port=45321
--webserver_interface=127.3.217.2
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:08:31.579578 18304 log.cc:824] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c: Log is configured to *not* fsync() on all Append() calls
Sep 24 07:08:31 dist-test-slave-2p7j krb5kdc[17643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233711, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:08:31.587525 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:31.568793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:58389 (local address 127.3.217.62:42227)
0924 07:08:31.569119 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:31.569136 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:31.569963 (+ 827us) server_negotiation.cc:407] Connection header received
0924 07:08:31.571299 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:31.571310 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:31.571724 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:31.572086 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:31.573500 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.574951 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.577009 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.577957 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.578181 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:31.581373 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:31.581380 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:31.581396 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:31.581430 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:31.583393 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:31.583995 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:31.584001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:31.584006 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:31.584171 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:31.584538 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:31.584542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:31.584544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:31.585020 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:31.585158 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:31.586237 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0924 07:08:31.586576 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":98,"threads_started":1}
I20260924 07:08:31.588045 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:31.568981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.1:58389)
0924 07:08:31.569698 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:31.569751 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:31.570812 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:31.572179 (+ 1367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:31.572188 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:31.572708 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:31.573289 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:31.573307 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.575145 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.575157 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:31.576770 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:31.576790 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.578184 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.578190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:31.578450 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:31.579144 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:31.579165 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:31.581140 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:31.583548 (+ 2408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:31.583554 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:31.583566 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:31.583878 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:31.584271 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:31.584274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:31.584276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:31.584426 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:31.585313 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:31.585322 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:31.585768 (+ 446us) client_negotiation.cc:769] Sending connection context
0924 07:08:31.586067 (+ 299us) client_negotiation.cc:240] Negotiation successful
0924 07:08:31.586518 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":225,"threads_started":1}
I20260924 07:08:31.588671 18304 tablet_bootstrap.cc:492] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:08:31.589062 18304 tablet_bootstrap.cc:492] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c: Bootstrap complete.
I20260924 07:08:31.590364 18304 ts_tablet_manager.cc:1397] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c: Time spent bootstrapping tablet: real 0.034s user 0.025s sys 0.008s
I20260924 07:08:31.590781 18412 heartbeater.cc:347] Connected to a master server at 127.3.217.62:42227
I20260924 07:08:31.591008 18412 heartbeater.cc:464] Registering TS with master...
I20260924 07:08:31.591585 18412 heartbeater.cc:511] Master 127.3.217.62:42227 requested a full tablet report, sending...
I20260924 07:08:31.593521 18304 raft_consensus.cc:348] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:31.593924 18304 raft_consensus.cc:729] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c915cd7cabc242f9884a2dfc5c20106c, State: Initialized, Role: FOLLOWER
I20260924 07:08:31.594354 18304 consensus_queue.cc:260] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:31.594431 18224 ts_manager.cc:194] Registered new tserver with Master: c915cd7cabc242f9884a2dfc5c20106c (127.3.217.1:38149)
I20260924 07:08:31.595214 18304 ts_tablet_manager.cc:1428] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:08:31.599179 18224 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.1@KRBTEST.COM'} at 127.3.217.1:58389
I20260924 07:08:31.601482 18412 heartbeater.cc:503] Master 127.3.217.62:42227 was elected leader, sending a full tablet report...
Sep 24 07:08:31 dist-test-slave-2p7j krb5kdc[17643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233711, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for kudu/127.3.217.1@KRBTEST.COM
I20260924 07:08:31.611435 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:31.598605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51838 (local address 127.3.217.1:38149)
0924 07:08:31.599289 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:31.599301 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:31.599322 (+ 21us) server_negotiation.cc:407] Connection header received
0924 07:08:31.599384 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:31.599387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:31.599577 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:31.599694 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:31.600427 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.601497 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.602709 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.603563 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.603670 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:31.606599 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:31.606615 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:31.606632 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:31.606668 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:31.608887 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:31.609547 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:31.609554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:31.609555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:31.609663 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:31.610036 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:31.610039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:31.610041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:31.610438 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:31.610563 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:31.611058 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 07:08:31.611273 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":118,"threads_started":1}
I20260924 07:08:31.611614 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:31.598532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:38149 (local address 127.0.0.1:51838)
0924 07:08:31.598952 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:31.598993 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:31.599124 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:31.599767 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:31.599783 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:31.600189 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:31.600320 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:31.600328 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.601637 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.601640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:31.602561 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:31.602574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.603696 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.603700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:31.603810 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:31.604373 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:31.604385 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:31.606381 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:31.609049 (+ 2668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:31.609058 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:31.609063 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:31.609384 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:31.609779 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:31.609782 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:31.609784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:31.609915 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:31.610584 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:31.610589 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:31.610728 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 07:08:31.610890 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 07:08:31.611106 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":106,"threads_started":1}
I20260924 07:08:31.612599 18346 tablet_service.cc:1514] Processing DeleteTablet for tablet 76f38037b25d41c78c0f41555d27b0ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:08:31 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:51838
I20260924 07:08:31.613451 18421 tablet_replica.cc:330] stopping tablet replica
I20260924 07:08:31.613762 18421 raft_consensus.cc:2229] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:08:31.614032 18421 raft_consensus.cc:2258] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:08:31.615119 18421 ts_tablet_manager.cc:1905] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:08:31.620149 18421 ts_tablet_manager.cc:1918] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:08:31.620319 18421 log.cc:1196] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/wal/wals/76f38037b25d41c78c0f41555d27b0ee
I20260924 07:08:31.620676 18421 ts_tablet_manager.cc:1939] T 76f38037b25d41c78c0f41555d27b0ee P c915cd7cabc242f9884a2dfc5c20106c: Deleting consensus metadata
I20260924 07:08:31.622967 18209 catalog_manager.cc:4841] TS c915cd7cabc242f9884a2dfc5c20106c (127.3.217.1:38149): tablet 76f38037b25d41c78c0f41555d27b0ee (table test-table [id=c27f1f55be904c80be17fe81c8ff6a19]) successfully deleted
W20260924 07:08:31.700631 18416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:31.700870 18416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:31.700909 18416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:31.704831 18416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:31.704887 18416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:31.704943 18416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:08:31.709903 18416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:35231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=45321
--tserver_master_addrs=127.3.217.62:42227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:31.710866 18416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:31.712172 18416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:31.723176 18429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:31.723328 18427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:31.723258 18426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:31.723649 18416 server_base.cc:1034] running on GCE node
I20260924 07:08:31.723995 18416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:31.724666 18416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:31.725850 18416 hybrid_clock.cc:648] HybridClock initialized: now 1790233711725802 us; error 66 us; skew 500 ppm
Sep 24 07:08:31 dist-test-slave-2p7j krb5kdc[17643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233711, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:31.728556 18416 init.cc:377] Logged in from keytab as kudu/127.3.217.2@KRBTEST.COM (short username kudu)
I20260924 07:08:31.729336 18416 webserver.cc:460] Webserver started at http://127.3.217.2:45321/ using document root <none> and password file <none>
I20260924 07:08:31.729825 18416 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:31.729909 18416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:08:31.733572 18416 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:08:31.735662 18436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:31.737454 18416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:08:31.737648 18416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "55769f9844444e4a9ce9d7062d8efcf4"
format_stamp: "Formatted at 2026-09-24 07:08:30 on dist-test-slave-2p7j"
server_key: "de1b90e3e363e91ee6b76fa8752cf7c2"
server_key_iv: "992c96dbd200865d84f9e8f65af179ea"
server_key_version: "encryptionkey@0"
I20260924 07:08:31.738400 18416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:31.763571 18416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:31.766889 18416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:31.767071 18416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:31.767661 18416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:08:31.768787 18443 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 07:08:31.774080 18416 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 07:08:31.774202 18416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260924 07:08:31.774340 18416 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 07:08:31.776664 18416 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 07:08:31.776722 18416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.001s
I20260924 07:08:31.777061 18443 tablet_bootstrap.cc:492] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4: Bootstrap starting.
I20260924 07:08:31.782500 18443 log.cc:824] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:08:31.787570 18416 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:35231
I20260924 07:08:31.787653 18551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:35231 every 8 connection(s)
I20260924 07:08:31.788375 18416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:08:31.790637 18443 tablet_bootstrap.cc:492] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:08:31.790987 18443 tablet_bootstrap.cc:492] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4: Bootstrap complete.
I20260924 07:08:31.791322 18443 ts_tablet_manager.cc:1397] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.000s
I20260924 07:08:31.793869 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 18416
I20260924 07:08:31.794402 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:35729
--local_ip_for_outbound_sockets=127.3.217.3
--tserver_master_addrs=127.3.217.62:42227
--webserver_port=41947
--webserver_interface=127.3.217.3
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:08:31.794158 18443 raft_consensus.cc:348] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:31.794673 18443 raft_consensus.cc:729] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55769f9844444e4a9ce9d7062d8efcf4, State: Initialized, Role: FOLLOWER
I20260924 07:08:31.795188 18443 consensus_queue.cc:260] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:31.796154 18443 ts_tablet_manager.cc:1428] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
Sep 24 07:08:31 dist-test-slave-2p7j krb5kdc[17643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233711, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:08:31.804237 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:31.790188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:37253 (local address 127.3.217.62:42227)
0924 07:08:31.790332 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:31.790337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:31.790970 (+ 633us) server_negotiation.cc:407] Connection header received
0924 07:08:31.792012 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:31.792016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:31.792074 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:31.792179 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:31.794010 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.794767 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.796495 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.796780 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.796916 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:31.800094 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:31.800098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:31.800101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:31.800133 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:31.801984 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:31.802517 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:31.802521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:31.802523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:31.802623 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:31.802973 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:31.802977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:31.802978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:31.803233 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:31.803329 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:31.803904 (+ 575us) server_negotiation.cc:299] Negotiation successful
0924 07:08:31.804050 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:08:31.804968 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:31.790310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.2:37253)
0924 07:08:31.790805 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:31.790853 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:31.791791 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:31.792520 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:31.792532 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:31.793089 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:31.793749 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:31.793770 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.794960 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.794964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:31.796322 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:31.796339 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.796926 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.796930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:31.797233 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:31.798462 (+ 1229us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:31.798481 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:31.799889 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:31.802131 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:31.802136 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:31.802149 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:31.802404 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:31.802722 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:31.802725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:31.802726 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:31.802874 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:31.803346 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:31.803351 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:31.803660 (+ 309us) client_negotiation.cc:769] Sending connection context
0924 07:08:31.803814 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 07:08:31.804082 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":123,"threads_started":1}
I20260924 07:08:31.806129 18552 heartbeater.cc:347] Connected to a master server at 127.3.217.62:42227
I20260924 07:08:31.806340 18552 heartbeater.cc:464] Registering TS with master...
I20260924 07:08:31.806857 18552 heartbeater.cc:511] Master 127.3.217.62:42227 requested a full tablet report, sending...
I20260924 07:08:31.808523 18224 ts_manager.cc:194] Registered new tserver with Master: 55769f9844444e4a9ce9d7062d8efcf4 (127.3.217.2:35231)
I20260924 07:08:31.809847 18224 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.2@KRBTEST.COM'} at 127.3.217.2:37253
I20260924 07:08:31.812152 18552 heartbeater.cc:503] Master 127.3.217.62:42227 was elected leader, sending a full tablet report...
Sep 24 07:08:31 dist-test-slave-2p7j krb5kdc[17643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233711, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for kudu/127.3.217.2@KRBTEST.COM
I20260924 07:08:31.820062 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:31.809454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:35231 (local address 127.0.0.1:43784)
0924 07:08:31.809579 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:31.809591 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:31.809665 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:31.810533 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:31.810536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:31.810715 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:31.810835 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:31.810844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.812371 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.812375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:31.813114 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:31.813124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.813924 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.813928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:31.814001 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:31.814491 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:31.814505 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:31.816017 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:31.818525 (+ 2508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:31.818527 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:31.818529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:31.818730 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:31.819030 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:31.819033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:31.819034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:31.819122 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:31.819606 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:31.819609 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:31.819689 (+ 80us) client_negotiation.cc:769] Sending connection context
0924 07:08:31.819800 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 07:08:31.819946 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 07:08:31.820194 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:31.809828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43784 (local address 127.3.217.2:35231)
0924 07:08:31.810119 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:31.810130 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:31.810152 (+ 22us) server_negotiation.cc:407] Connection header received
0924 07:08:31.810210 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:31.810215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:31.810413 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:31.810572 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:31.810940 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.812255 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.813232 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.813787 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.813893 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:31.816149 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:31.816158 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:31.816171 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:31.816196 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:31.818387 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:31.818828 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:31.818833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:31.818835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:31.818922 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:31.819208 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:31.819211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:31.819213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:31.819478 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:31.819632 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:31.819871 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 07:08:31.820029 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":111,"threads_started":1}
I20260924 07:08:31.821116 18485 tablet_service.cc:1514] Processing DeleteTablet for tablet 76f38037b25d41c78c0f41555d27b0ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:08:31 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:43784
I20260924 07:08:31.822064 18558 tablet_replica.cc:330] stopping tablet replica
I20260924 07:08:31.822315 18558 raft_consensus.cc:2229] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:08:31.822533 18558 raft_consensus.cc:2258] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:08:31.823372 18558 ts_tablet_manager.cc:1905] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:08:31.826910 18558 ts_tablet_manager.cc:1918] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:08:31.826997 18558 log.cc:1196] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/76f38037b25d41c78c0f41555d27b0ee
I20260924 07:08:31.827235 18558 ts_tablet_manager.cc:1939] T 76f38037b25d41c78c0f41555d27b0ee P 55769f9844444e4a9ce9d7062d8efcf4: Deleting consensus metadata
I20260924 07:08:31.828953 18210 catalog_manager.cc:4841] TS 55769f9844444e4a9ce9d7062d8efcf4 (127.3.217.2:35231): tablet 76f38037b25d41c78c0f41555d27b0ee (table test-table [id=c27f1f55be904c80be17fe81c8ff6a19]) successfully deleted
W20260924 07:08:31.916713 18556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:31.916967 18556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:31.917030 18556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:31.921474 18556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:31.921537 18556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:31.921618 18556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:08:31.926965 18556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:35729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=41947
--tserver_master_addrs=127.3.217.62:42227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:31.927940 18556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:31.928869 18556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:31.939435 18563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:31.939435 18564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:31.939477 18566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:31.940006 18556 server_base.cc:1034] running on GCE node
I20260924 07:08:31.940399 18556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:31.941027 18556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:31.942234 18556 hybrid_clock.cc:648] HybridClock initialized: now 1790233711942216 us; error 32 us; skew 500 ppm
Sep 24 07:08:31 dist-test-slave-2p7j krb5kdc[17643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233711, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:31.945647 18556 init.cc:377] Logged in from keytab as kudu/127.3.217.3@KRBTEST.COM (short username kudu)
I20260924 07:08:31.946513 18556 webserver.cc:460] Webserver started at http://127.3.217.3:41947/ using document root <none> and password file <none>
I20260924 07:08:31.947005 18556 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:31.947077 18556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:08:31.951256 18556 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:08:31.953923 18573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:31.955298 18556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:08:31.955410 18556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "8c837b29bb524c38890cc1328b014f4b"
format_stamp: "Formatted at 2026-09-24 07:08:30 on dist-test-slave-2p7j"
server_key: "ea45bdf74a71496307e61bf091cc8677"
server_key_iv: "b999b8530ece4a1072438bd5472966f7"
server_key_version: "encryptionkey@0"
I20260924 07:08:31.955988 18556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:31.967872 18556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:31.971828 18556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:31.972002 18556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:31.972733 18556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:08:31.973549 18556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:08:31.973600 18556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:31.973636 18556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:08:31.973652 18556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:31.982770 18556 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:35729
I20260924 07:08:31.982817 18686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:35729 every 8 connection(s)
I20260924 07:08:31.983850 18556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:08:31.993479 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 18556
Sep 24 07:08:31 dist-test-slave-2p7j krb5kdc[17643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233711, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:08:31.999697 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:31.986120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:51537 (local address 127.3.217.62:42227)
0924 07:08:31.986303 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:31.986307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:31.987674 (+ 1367us) server_negotiation.cc:407] Connection header received
0924 07:08:31.988779 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:31.988782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:31.988837 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:31.988950 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:31.990709 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.991316 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.993072 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.993635 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.993758 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:31.996285 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:31.996288 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:31.996290 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:31.996311 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:31.997768 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:31.998261 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:31.998264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:31.998265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:31.998333 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:31.998643 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:31.998646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:31.998659 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:31.998816 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:31.998895 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:31.999402 (+ 507us) server_negotiation.cc:299] Negotiation successful
0924 07:08:31.999543 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 07:08:32.000599 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:31.986457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.3:51537)
0924 07:08:31.987428 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:31.987488 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:31.988531 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:31.989253 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:31.989262 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:31.989786 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:31.990475 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:31.990498 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.991494 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.991534 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:31.992870 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:31.992889 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.993810 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.993813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:31.994002 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:31.994643 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:31.994661 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:31.996125 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:31.997872 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:31.997878 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:31.997894 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:31.998160 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:31.998412 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:31.998415 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:31.998417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:31.998550 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:31.998915 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:31.998921 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:31.999175 (+ 254us) client_negotiation.cc:769] Sending connection context
0924 07:08:31.999331 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 07:08:31.999696 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":109,"threads_started":1}
I20260924 07:08:32.002206 18687 heartbeater.cc:347] Connected to a master server at 127.3.217.62:42227
I20260924 07:08:32.002470 18687 heartbeater.cc:464] Registering TS with master...
I20260924 07:08:32.002852 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:31.994209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38842 (local address 127.3.217.62:42227)
0924 07:08:31.994437 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:31.994441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:31.994610 (+ 169us) server_negotiation.cc:407] Connection header received
0924 07:08:31.994736 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:31.994739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:31.994777 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:31.994849 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:31.995556 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.996136 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.996957 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:31.997185 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:31.997267 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:31.999076 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:31.999080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:31.999082 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:31.999104 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:32.000487 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:32.000940 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:32.000943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:32.000945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:32.001018 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:32.001497 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:32.001503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:32.001505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:32.001838 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:32.002043 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:32.002526 (+ 483us) server_negotiation.cc:299] Negotiation successful
0924 07:08:32.002702 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":70,"threads_started":1}
I20260924 07:08:32.003047 18687 heartbeater.cc:511] Master 127.3.217.62:42227 requested a full tablet report, sending...
I20260924 07:08:32.004029 18223 ts_manager.cc:194] Registered new tserver with Master: 8c837b29bb524c38890cc1328b014f4b (127.3.217.3:35729)
I20260924 07:08:32.004654 18223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.3@KRBTEST.COM'} at 127.3.217.3:51537
I20260924 07:08:32.016820 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:08:32.021274 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:32.017681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38852 (local address 127.3.217.62:42227)
0924 07:08:32.017818 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:32.017822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:32.017959 (+ 137us) server_negotiation.cc:407] Connection header received
0924 07:08:32.018113 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:32.018116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:32.018209 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:32.018294 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:08:32.018520 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:32.019033 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:32.019973 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:32.020197 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:32.020281 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:32.020550 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:08:32.020925 (+ 375us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:08:32.021034 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:32.021098 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 07:08:32.021155 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 07:08:32.021835 18223 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:08:32.024983 18223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:08:32.034215 18485 tablet_service.cc:1467] Processing CreateTablet for tablet 9da85943de164c978ced1f9a6398b3ea (DEFAULT_TABLE table=test-table [id=dadba52f00ef456680bdc11d609c0e2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:08:32.034286 18346 tablet_service.cc:1467] Processing CreateTablet for tablet 9da85943de164c978ced1f9a6398b3ea (DEFAULT_TABLE table=test-table [id=dadba52f00ef456680bdc11d609c0e2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:08:32.034767 18485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9da85943de164c978ced1f9a6398b3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:08:32.034863 18346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9da85943de164c978ced1f9a6398b3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:08:32.037322 18304 tablet_bootstrap.cc:492] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c: Bootstrap starting.
I20260924 07:08:32.037403 18443 tablet_bootstrap.cc:492] T 9da85943de164c978ced1f9a6398b3ea P 55769f9844444e4a9ce9d7062d8efcf4: Bootstrap starting.
I20260924 07:08:32.038144 18304 tablet_bootstrap.cc:654] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c: Neither blocks nor log segments found. Creating new log.
I20260924 07:08:32.038285 18443 tablet_bootstrap.cc:654] T 9da85943de164c978ced1f9a6398b3ea P 55769f9844444e4a9ce9d7062d8efcf4: Neither blocks nor log segments found. Creating new log.
I20260924 07:08:32.039436 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:32.033336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:35729 (local address 127.0.0.1:50392)
0924 07:08:32.033484 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:32.033499 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:32.033572 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:32.034565 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:32.034568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:32.034585 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:08:32.034681 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:32.034687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:32.036374 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:32.036378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:32.037539 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:32.037550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:32.039049 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:32.039052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:32.039142 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:32.039192 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 07:08:32.039247 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 07:08:32.039321 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 07:08:32.039610 18304 tablet_bootstrap.cc:492] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c: No bootstrap required, opened a new log
I20260924 07:08:32.039695 18304 ts_tablet_manager.cc:1397] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260924 07:08:32.039718 18443 tablet_bootstrap.cc:492] T 9da85943de164c978ced1f9a6398b3ea P 55769f9844444e4a9ce9d7062d8efcf4: No bootstrap required, opened a new log
I20260924 07:08:32.039780 18443 ts_tablet_manager.cc:1397] T 9da85943de164c978ced1f9a6398b3ea P 55769f9844444e4a9ce9d7062d8efcf4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:08:32.040009 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:32.033626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50392 (local address 127.3.217.3:35729)
0924 07:08:32.034008 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:32.034021 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:32.034047 (+ 26us) server_negotiation.cc:407] Connection header received
0924 07:08:32.034122 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:32.034127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:32.034369 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:32.034830 (+ 461us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:08:32.035128 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:32.036231 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:32.037832 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:32.038919 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:32.039627 (+ 708us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:32.039769 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:32.039819 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 07:08:32.039905 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":151,"threads_started":1}
I20260924 07:08:32.040145 18443 raft_consensus.cc:348] T 9da85943de164c978ced1f9a6398b3ea P 55769f9844444e4a9ce9d7062d8efcf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:32.040123 18304 raft_consensus.cc:348] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:32.040243 18443 raft_consensus.cc:374] T 9da85943de164c978ced1f9a6398b3ea P 55769f9844444e4a9ce9d7062d8efcf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:08:32.040243 18304 raft_consensus.cc:374] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:08:32.040287 18443 raft_consensus.cc:729] T 9da85943de164c978ced1f9a6398b3ea P 55769f9844444e4a9ce9d7062d8efcf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55769f9844444e4a9ce9d7062d8efcf4, State: Initialized, Role: FOLLOWER
I20260924 07:08:32.040287 18304 raft_consensus.cc:729] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c915cd7cabc242f9884a2dfc5c20106c, State: Initialized, Role: FOLLOWER
I20260924 07:08:32.040400 18304 consensus_queue.cc:260] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:32.040400 18443 consensus_queue.cc:260] T 9da85943de164c978ced1f9a6398b3ea P 55769f9844444e4a9ce9d7062d8efcf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:32.040642 18304 ts_tablet_manager.cc:1428] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 07:08:32.040642 18443 ts_tablet_manager.cc:1428] T 9da85943de164c978ced1f9a6398b3ea P 55769f9844444e4a9ce9d7062d8efcf4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:08:32.042912 18621 tablet_service.cc:1467] Processing CreateTablet for tablet 9da85943de164c978ced1f9a6398b3ea (DEFAULT_TABLE table=test-table [id=dadba52f00ef456680bdc11d609c0e2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:08:32.043885 18621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9da85943de164c978ced1f9a6398b3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:08:32.049996 18699 tablet_bootstrap.cc:492] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b: Bootstrap starting.
I20260924 07:08:32.052372 18699 tablet_bootstrap.cc:654] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b: Neither blocks nor log segments found. Creating new log.
I20260924 07:08:32.053467 18699 log.cc:824] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b: Log is configured to *not* fsync() on all Append() calls
I20260924 07:08:32.055840 18699 tablet_bootstrap.cc:492] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b: No bootstrap required, opened a new log
I20260924 07:08:32.056026 18699 ts_tablet_manager.cc:1397] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:08:32.058676 18699 raft_consensus.cc:348] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:32.058889 18699 raft_consensus.cc:374] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:08:32.058949 18699 raft_consensus.cc:729] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c837b29bb524c38890cc1328b014f4b, State: Initialized, Role: FOLLOWER
I20260924 07:08:32.059381 18699 consensus_queue.cc:260] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:32.060240 18687 heartbeater.cc:503] Master 127.3.217.62:42227 was elected leader, sending a full tablet report...
I20260924 07:08:32.060269 18699 ts_tablet_manager.cc:1428] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
W20260924 07:08:32.067703 18413 tablet.cc:2369] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:08:32.183246 18701 raft_consensus.cc:482] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:08:32.183482 18701 raft_consensus.cc:504] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:32.184790 18701 leader_election.cc:288] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c915cd7cabc242f9884a2dfc5c20106c (127.3.217.1:38149), 55769f9844444e4a9ce9d7062d8efcf4 (127.3.217.2:35231)
I20260924 07:08:32.189716 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:32.185044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:38149 (local address 127.3.217.3:40715)
0924 07:08:32.185554 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:32.185569 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:32.185665 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:32.185957 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:32.185960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:32.185978 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:08:32.186092 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:32.186099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:32.187160 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:32.187163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:32.188206 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:32.188217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:32.189061 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:32.189065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:32.189294 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:32.189318 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 07:08:32.189401 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:08:32.189483 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":86,"thread_start_us":106,"threads_started":1}
I20260924 07:08:32.189730 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:32.185232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:42947 (local address 127.3.217.2:35231)
0924 07:08:32.185388 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:32.185395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:32.185413 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:08:32.185470 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:32.185474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:32.185540 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:32.185654 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:08:32.185957 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:32.186794 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:32.188397 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:32.188909 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:32.189024 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:32.189188 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:32.189473 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 07:08:32.189560 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:08:32.189854 18703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:32.185216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:40715 (local address 127.3.217.1:38149)
0924 07:08:32.185456 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:32.185460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:32.185645 (+ 185us) server_negotiation.cc:407] Connection header received
0924 07:08:32.185776 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:32.185779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:32.185841 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:32.185934 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:08:32.186330 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:32.186979 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:32.188354 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:32.188926 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:32.189040 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:32.189238 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:32.189444 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 07:08:32.189536 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":67,"threads_started":1}
I20260924 07:08:32.189883 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:32.185021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:35231 (local address 127.3.217.3:42947)
0924 07:08:32.185160 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:32.185175 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:32.185258 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:32.185678 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:32.185681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:32.185700 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:08:32.185819 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:32.185826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:32.187019 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:32.187024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:32.188221 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:32.188236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:32.189063 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:32.189067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:32.189331 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:32.189351 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 07:08:32.189434 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:08:32.189533 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":178}
I20260924 07:08:32.190393 18366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da85943de164c978ced1f9a6398b3ea" candidate_uuid: "8c837b29bb524c38890cc1328b014f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c915cd7cabc242f9884a2dfc5c20106c" is_pre_election: true
I20260924 07:08:32.190586 18506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da85943de164c978ced1f9a6398b3ea" candidate_uuid: "8c837b29bb524c38890cc1328b014f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55769f9844444e4a9ce9d7062d8efcf4" is_pre_election: true
I20260924 07:08:32.190735 18366 raft_consensus.cc:2454] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c837b29bb524c38890cc1328b014f4b in term 0.
I20260924 07:08:32.190861 18506 raft_consensus.cc:2454] T 9da85943de164c978ced1f9a6398b3ea P 55769f9844444e4a9ce9d7062d8efcf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c837b29bb524c38890cc1328b014f4b in term 0.
I20260924 07:08:32.191288 18576 leader_election.cc:302] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55769f9844444e4a9ce9d7062d8efcf4, 8c837b29bb524c38890cc1328b014f4b; no voters:
I20260924 07:08:32.191548 18701 raft_consensus.cc:2781] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:08:32.191637 18701 raft_consensus.cc:482] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:08:32.191689 18701 raft_consensus.cc:3037] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:08:32.192723 18701 raft_consensus.cc:504] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:32.193056 18701 leader_election.cc:288] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b [CANDIDATE]: Term 1 election: Requested vote from peers c915cd7cabc242f9884a2dfc5c20106c (127.3.217.1:38149), 55769f9844444e4a9ce9d7062d8efcf4 (127.3.217.2:35231)
I20260924 07:08:32.193369 18506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da85943de164c978ced1f9a6398b3ea" candidate_uuid: "8c837b29bb524c38890cc1328b014f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55769f9844444e4a9ce9d7062d8efcf4"
I20260924 07:08:32.193423 18366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da85943de164c978ced1f9a6398b3ea" candidate_uuid: "8c837b29bb524c38890cc1328b014f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c915cd7cabc242f9884a2dfc5c20106c"
I20260924 07:08:32.193476 18506 raft_consensus.cc:3037] T 9da85943de164c978ced1f9a6398b3ea P 55769f9844444e4a9ce9d7062d8efcf4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:08:32.193571 18366 raft_consensus.cc:3037] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:08:32.194429 18506 raft_consensus.cc:2454] T 9da85943de164c978ced1f9a6398b3ea P 55769f9844444e4a9ce9d7062d8efcf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c837b29bb524c38890cc1328b014f4b in term 1.
I20260924 07:08:32.194578 18366 raft_consensus.cc:2454] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c837b29bb524c38890cc1328b014f4b in term 1.
I20260924 07:08:32.194661 18576 leader_election.cc:302] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55769f9844444e4a9ce9d7062d8efcf4, 8c837b29bb524c38890cc1328b014f4b; no voters:
I20260924 07:08:32.194804 18701 raft_consensus.cc:2781] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:08:32.195032 18701 raft_consensus.cc:686] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b [term 1 LEADER]: Becoming Leader. State: Replica: 8c837b29bb524c38890cc1328b014f4b, State: Running, Role: LEADER
I20260924 07:08:32.195322 18701 consensus_queue.cc:237] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:32.198652 18223 catalog_manager.cc:5495] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c837b29bb524c38890cc1328b014f4b (127.3.217.3). New cstate: current_term: 1 leader_uuid: "8c837b29bb524c38890cc1328b014f4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } health_report { overall_health: UNKNOWN } } }
I20260924 07:08:32.231134 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:32.227109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50398 (local address 127.3.217.3:35729)
0924 07:08:32.227228 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:32.227231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:32.227278 (+ 47us) server_negotiation.cc:407] Connection header received
0924 07:08:32.227587 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:32.227590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:32.227653 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:32.227747 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:08:32.228016 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:32.228507 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:32.229645 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:32.229990 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:32.230077 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:32.230347 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:08:32.230764 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:08:32.230846 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:32.230957 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 07:08:32.231008 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 07:08:32.234760 18688 tablet.cc:2369] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:08:32.237375 18506 raft_consensus.cc:1261] T 9da85943de164c978ced1f9a6398b3ea P 55769f9844444e4a9ce9d7062d8efcf4 [term 1 FOLLOWER]: Refusing update from remote peer 8c837b29bb524c38890cc1328b014f4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:08:32.237376 18366 raft_consensus.cc:1261] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Refusing update from remote peer 8c837b29bb524c38890cc1328b014f4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:08:32.237957 18704 consensus_queue.cc:1035] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:08:32.238149 18701 consensus_queue.cc:1035] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:08:32.243988 18706 mvcc.cc:205] Tried to move back new op lower bound from 7332797285318008832 to 7332797285154717696. Current Snapshot: MvccSnapshot[applied={T|T < 7332797285318008832}]
I20260924 07:08:32.253086 18224 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:08:32.253279 18224 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:08:32.256158 18224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc: Sending DeleteTablet for 3 replicas of tablet 9da85943de164c978ced1f9a6398b3ea
I20260924 07:08:32.256906 18621 tablet_service.cc:1514] Processing DeleteTablet for tablet 9da85943de164c978ced1f9a6398b3ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:08:32 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:50392
I20260924 07:08:32.257002 18346 tablet_service.cc:1514] Processing DeleteTablet for tablet 9da85943de164c978ced1f9a6398b3ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:08:32 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:51838
I20260924 07:08:32.257277 18485 tablet_service.cc:1514] Processing DeleteTablet for tablet 9da85943de164c978ced1f9a6398b3ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:08:32 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:43784
I20260924 07:08:32.257313 18716 tablet_replica.cc:330] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c: stopping tablet replica
I20260924 07:08:32.257418 18716 raft_consensus.cc:2229] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:08:32.257449 18558 tablet_replica.cc:330] T 9da85943de164c978ced1f9a6398b3ea P 55769f9844444e4a9ce9d7062d8efcf4: stopping tablet replica
I20260924 07:08:32.257529 18558 raft_consensus.cc:2229] T 9da85943de164c978ced1f9a6398b3ea P 55769f9844444e4a9ce9d7062d8efcf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:08:32.257544 18716 raft_consensus.cc:2258] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:08:32.257632 18558 raft_consensus.cc:2258] T 9da85943de164c978ced1f9a6398b3ea P 55769f9844444e4a9ce9d7062d8efcf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:08:32.258178 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 18262
I20260924 07:08:32.258981 18558 ts_tablet_manager.cc:1905] T 9da85943de164c978ced1f9a6398b3ea P 55769f9844444e4a9ce9d7062d8efcf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:08:32.262729 18558 ts_tablet_manager.cc:1918] T 9da85943de164c978ced1f9a6398b3ea P 55769f9844444e4a9ce9d7062d8efcf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:08:32.262809 18558 log.cc:1196] T 9da85943de164c978ced1f9a6398b3ea P 55769f9844444e4a9ce9d7062d8efcf4: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/9da85943de164c978ced1f9a6398b3ea
I20260924 07:08:32.263083 18558 ts_tablet_manager.cc:1939] T 9da85943de164c978ced1f9a6398b3ea P 55769f9844444e4a9ce9d7062d8efcf4: Deleting consensus metadata
I20260924 07:08:32.263454 18717 tablet_replica.cc:330] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b: stopping tablet replica
I20260924 07:08:32.263779 18717 raft_consensus.cc:2229] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:08:32.264014 18210 catalog_manager.cc:4841] TS 55769f9844444e4a9ce9d7062d8efcf4 (127.3.217.2:35231): tablet 9da85943de164c978ced1f9a6398b3ea (table test-table [id=dadba52f00ef456680bdc11d609c0e2b]) successfully deleted
I20260924 07:08:32.264161 18717 raft_consensus.cc:2258] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:08:32.265419 18717 ts_tablet_manager.cc:1905] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 07:08:32.267673 18209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.217.1:38149 (error 108)
I20260924 07:08:32.268234 18717 ts_tablet_manager.cc:1918] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:08:32.268348 18717 log.cc:1196] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/9da85943de164c978ced1f9a6398b3ea
I20260924 07:08:32.268698 18717 ts_tablet_manager.cc:1939] T 9da85943de164c978ced1f9a6398b3ea P 8c837b29bb524c38890cc1328b014f4b: Deleting consensus metadata
W20260924 07:08:32.268908 18419 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:08:32.268615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:38149 (local address 127.0.0.1:51850)
0924 07:08:32.268747 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:32.268819 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.1:38149: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 07:08:32.269043 18209 catalog_manager.cc:4584] TS c915cd7cabc242f9884a2dfc5c20106c (127.3.217.1:38149): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9da85943de164c978ced1f9a6398b3ea: Network error: Client connection negotiation failed: client connection to 127.3.217.1:38149: connect: Connection reset by peer (error 104)
I20260924 07:08:32.269821 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 18416
I20260924 07:08:32.269838 18211 catalog_manager.cc:4841] TS 8c837b29bb524c38890cc1328b014f4b (127.3.217.3:35729): tablet 9da85943de164c978ced1f9a6398b3ea (table test-table [id=dadba52f00ef456680bdc11d609c0e2b]) successfully deleted
I20260924 07:08:32.276130 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 18556
I20260924 07:08:32.283046 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 18191
I20260924 07:08:32.290958 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:42227
--webserver_interface=127.3.217.62
--webserver_port=33551
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:42227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:08:32.401717 18719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:32.401996 18719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:32.402042 18719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:32.406260 18719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:08:32.406327 18719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:32.406349 18719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:32.406375 18719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:08:32.406394 18719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:08:32.411688 18719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:42227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:42227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:32.412724 18719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:32.413718 18719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:32.422606 18725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:32.422597 18727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:32.422792 18724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:32.422950 18719 server_base.cc:1034] running on GCE node
I20260924 07:08:32.423568 18719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:32.424650 18719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:32.425824 18719 hybrid_clock.cc:648] HybridClock initialized: now 1790233712425809 us; error 26 us; skew 500 ppm
Sep 24 07:08:32 dist-test-slave-2p7j krb5kdc[17643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233712, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:32.428651 18719 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:08:32.429596 18719 webserver.cc:460] Webserver started at http://127.3.217.62:33551/ using document root <none> and password file <none>
I20260924 07:08:32.430179 18719 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:32.430240 18719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:08:32.433869 18719 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:08:32.436220 18734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:32.437529 18719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:08:32.437803 18719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "08e2efa8c00d4b56a6006983b78126fc"
format_stamp: "Formatted at 2026-09-24 07:08:30 on dist-test-slave-2p7j"
server_key: "578f46ee9985c7a954001c6c61be9afe"
server_key_iv: "8e26479df6be03e65316169fb239dc05"
server_key_version: "encryptionkey@0"
I20260924 07:08:32.438462 18719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:32.460107 18719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:32.463642 18719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:32.463846 18719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:32.470969 18719 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:42227
I20260924 07:08:32.471246 18786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:42227 every 8 connection(s)
I20260924 07:08:32.472045 18719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:08:32.476472 18787 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:08:32.478202 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 18719
I20260924 07:08:32.478222 18787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc: Bootstrap starting.
I20260924 07:08:32.478799 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:38149
--local_ip_for_outbound_sockets=127.3.217.1
--tserver_master_addrs=127.3.217.62:42227
--webserver_port=33555
--webserver_interface=127.3.217.1
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:08:32.499140 18787 log.cc:824] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc: Log is configured to *not* fsync() on all Append() calls
I20260924 07:08:32.513713 18787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:08:32.514092 18787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc: Bootstrap complete.
I20260924 07:08:32.517901 18787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e2efa8c00d4b56a6006983b78126fc" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42227 } }
I20260924 07:08:32.518373 18787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08e2efa8c00d4b56a6006983b78126fc, State: Initialized, Role: FOLLOWER
I20260924 07:08:32.518975 18787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e2efa8c00d4b56a6006983b78126fc" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42227 } }
I20260924 07:08:32.519124 18787 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:08:32.519196 18787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:08:32.519279 18787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 2 FOLLOWER]: Advancing to term 3
I20260924 07:08:32.521198 18787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e2efa8c00d4b56a6006983b78126fc" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42227 } }
I20260924 07:08:32.521672 18787 leader_election.cc:302] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08e2efa8c00d4b56a6006983b78126fc; no voters:
I20260924 07:08:32.522194 18787 leader_election.cc:288] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 07:08:32.522337 18791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 3 FOLLOWER]: Leader election won for term 3
I20260924 07:08:32.522583 18791 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [term 3 LEADER]: Becoming Leader. State: Replica: 08e2efa8c00d4b56a6006983b78126fc, State: Running, Role: LEADER
I20260924 07:08:32.522964 18791 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e2efa8c00d4b56a6006983b78126fc" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42227 } }
I20260924 07:08:32.523893 18787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:08:32.524294 18793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08e2efa8c00d4b56a6006983b78126fc. Latest consensus state: current_term: 3 leader_uuid: "08e2efa8c00d4b56a6006983b78126fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e2efa8c00d4b56a6006983b78126fc" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42227 } } }
I20260924 07:08:32.524533 18793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [sys.catalog]: This master's current role is: LEADER
I20260924 07:08:32.524681 18792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "08e2efa8c00d4b56a6006983b78126fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e2efa8c00d4b56a6006983b78126fc" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42227 } } }
I20260924 07:08:32.524811 18792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08e2efa8c00d4b56a6006983b78126fc [sys.catalog]: This master's current role is: LEADER
I20260924 07:08:32.525065 18795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:08:32.529810 18795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:08:32.530652 18795 catalog_manager.cc:1259] Loaded cluster ID: 9f4bbd81a8e64ce89928c982f0c03895
I20260924 07:08:32.530712 18795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 07:08:32.531729 18795 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdedb7c09ad google::LogMessage::Fail() at ??:0
@ 0x7fdedb7c388a google::LogMessage::SendToLog() at ??:0
@ 0x7fdedb7c04de google::LogMessage::Flush() at ??:0
@ 0x7fdedb7c1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdee60fffdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fdee6100d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdee60fd579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fdee6141640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fdedda123d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdedda12c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fdedda143f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fdedda03512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdedf0476db start_thread at ??:0
@ 0x7fdeda2be61f clone at ??:0
W20260924 07:08:32.625810 18789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:32.626154 18789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:32.626227 18789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:32.632256 18789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:32.632341 18789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:32.632452 18789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:08:32.642169 18789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:38149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=33555
--tserver_master_addrs=127.3.217.62:42227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:32.643581 18789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:32.645098 18789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:32.656543 18829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:32.656559 18831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:32.656538 18828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:32.657130 18789 server_base.cc:1034] running on GCE node
I20260924 07:08:32.657554 18789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:32.658720 18789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:32.659981 18789 hybrid_clock.cc:648] HybridClock initialized: now 1790233712659937 us; error 48 us; skew 500 ppm
Sep 24 07:08:32 dist-test-slave-2p7j krb5kdc[17643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233712, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:32.664008 18789 init.cc:377] Logged in from keytab as kudu/127.3.217.1@KRBTEST.COM (short username kudu)
I20260924 07:08:32.664906 18789 webserver.cc:460] Webserver started at http://127.3.217.1:33555/ using document root <none> and password file <none>
I20260924 07:08:32.665427 18789 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:32.665488 18789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:08:32.669802 18789 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260924 07:08:32.672112 18838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:32.673724 18789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:08:32.673840 18789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "c915cd7cabc242f9884a2dfc5c20106c"
format_stamp: "Formatted at 2026-09-24 07:08:30 on dist-test-slave-2p7j"
server_key: "b14d3b7563b7989af5e2c0fdaf283802"
server_key_iv: "cc15ffdfe5c66ceb9fe7f060ffb2d594"
server_key_version: "encryptionkey@0"
I20260924 07:08:32.674383 18789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:32.697064 18789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:32.700428 18789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:32.700627 18789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:32.701192 18789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:08:32.702258 18845 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 07:08:32.707436 18789 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 07:08:32.707530 18789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260924 07:08:32.707638 18789 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 07:08:32.709224 18789 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 07:08:32.709267 18789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 07:08:32.709576 18845 tablet_bootstrap.cc:492] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c: Bootstrap starting.
I20260924 07:08:32.715991 18845 log.cc:824] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:08:32.719774 18789 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:38149
I20260924 07:08:32.720058 18953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:38149 every 8 connection(s)
I20260924 07:08:32.720646 18789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:08:32.724532 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 18789
I20260924 07:08:32.724886 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:35231
--local_ip_for_outbound_sockets=127.3.217.2
--tserver_master_addrs=127.3.217.62:42227
--webserver_port=45321
--webserver_interface=127.3.217.2
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:08:32.727275 18845 tablet_bootstrap.cc:492] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:08:32.727767 18845 tablet_bootstrap.cc:492] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c: Bootstrap complete.
I20260924 07:08:32.728140 18845 ts_tablet_manager.cc:1397] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.002s
I20260924 07:08:32.731278 18845 raft_consensus.cc:348] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:32.731863 18845 raft_consensus.cc:729] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c915cd7cabc242f9884a2dfc5c20106c, State: Initialized, Role: FOLLOWER
I20260924 07:08:32.732378 18845 consensus_queue.cc:260] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:32.733404 18845 ts_tablet_manager.cc:1428] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
W20260924 07:08:32.767132 18956 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:08:32.722417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.1:35497)
0924 07:08:32.722876 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:32.722925 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:32.723748 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:32.765853 (+ 42105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":110,"threads_started":1}
W20260924 07:08:32.767798 18954 heartbeater.cc:644] Failed to heartbeat to 127.3.217.62:42227 (0 consecutive failures): Network error: Failed to ping master at 127.3.217.62:42227: Client connection negotiation failed: client connection to 127.3.217.62:42227: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 07:08:32.768851 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:32.768457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.1:58455)
0924 07:08:32.768633 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:32.768748 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 07:08:32.769738 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:32.769482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.1:54573)
0924 07:08:32.769609 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:32.769669 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 07:08:32.770088 18954 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:08:32.844053 18957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:32.844306 18957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:32.844350 18957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:32.848309 18957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:32.848364 18957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:32.848428 18957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:08:32.853722 18957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:35231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=45321
--tserver_master_addrs=127.3.217.62:42227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:32.854621 18957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:32.855536 18957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:32.864904 18964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:32.864933 18966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:32.864914 18963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:32.865202 18957 server_base.cc:1034] running on GCE node
I20260924 07:08:32.865664 18957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:32.866405 18957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:32.867612 18957 hybrid_clock.cc:648] HybridClock initialized: now 1790233712867587 us; error 28 us; skew 500 ppm
Sep 24 07:08:32 dist-test-slave-2p7j krb5kdc[17643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233712, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:32.870821 18957 init.cc:377] Logged in from keytab as kudu/127.3.217.2@KRBTEST.COM (short username kudu)
I20260924 07:08:32.871704 18957 webserver.cc:460] Webserver started at http://127.3.217.2:45321/ using document root <none> and password file <none>
I20260924 07:08:32.872252 18957 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:32.872319 18957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:08:32.876626 18957 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:08:32.878818 18973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:32.880136 18957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:08:32.880246 18957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "55769f9844444e4a9ce9d7062d8efcf4"
format_stamp: "Formatted at 2026-09-24 07:08:30 on dist-test-slave-2p7j"
server_key: "de1b90e3e363e91ee6b76fa8752cf7c2"
server_key_iv: "992c96dbd200865d84f9e8f65af179ea"
server_key_version: "encryptionkey@0"
I20260924 07:08:32.880770 18957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:32.899335 18957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:32.903096 18957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:32.903285 18957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:32.903909 18957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:08:32.904721 18957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:08:32.904769 18957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:32.904822 18957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:08:32.904848 18957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:32.916481 18957 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:35231
I20260924 07:08:32.916520 19086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:35231 every 8 connection(s)
I20260924 07:08:32.917520 18957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:08:32.921752 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:32.920136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.2:45835)
0924 07:08:32.921003 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:32.921114 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":504,"threads_started":1}
W20260924 07:08:32.922254 19087 heartbeater.cc:644] Failed to heartbeat to 127.3.217.62:42227 (0 consecutive failures): Network error: Failed to ping master at 127.3.217.62:42227: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
I20260924 07:08:32.922313 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 18957
I20260924 07:08:32.922686 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:35729
--local_ip_for_outbound_sockets=127.3.217.3
--tserver_master_addrs=127.3.217.62:42227
--webserver_port=41947
--webserver_interface=127.3.217.3
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:08:32.922878 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:32.922637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.2:49895)
0924 07:08:32.922762 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:32.922815 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 07:08:32.923445 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:32.923251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.2:48647)
0924 07:08:32.923345 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:32.923389 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 07:08:32.923655 19087 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:08:33.043455 19090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:33.043741 19090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:33.043788 19090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:33.049258 19090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:33.049328 19090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:33.049387 19090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:08:33.055426 19090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:35729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=41947
--tserver_master_addrs=127.3.217.62:42227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:33.056422 19090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:33.057394 19090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:33.069386 19095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:33.069427 19096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:33.069798 19098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:33.070477 19090 server_base.cc:1034] running on GCE node
I20260924 07:08:33.071067 19090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:33.071796 19090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:33.073058 19090 hybrid_clock.cc:648] HybridClock initialized: now 1790233713073027 us; error 30 us; skew 500 ppm
Sep 24 07:08:33 dist-test-slave-2p7j krb5kdc[17643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233713, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:33.076267 19090 init.cc:377] Logged in from keytab as kudu/127.3.217.3@KRBTEST.COM (short username kudu)
I20260924 07:08:33.077168 19090 webserver.cc:460] Webserver started at http://127.3.217.3:41947/ using document root <none> and password file <none>
I20260924 07:08:33.077796 19090 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:33.077919 19090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:08:33.081763 19090 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:08:33.083997 19105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:33.085366 19090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:08:33.085479 19090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "8c837b29bb524c38890cc1328b014f4b"
format_stamp: "Formatted at 2026-09-24 07:08:30 on dist-test-slave-2p7j"
server_key: "ea45bdf74a71496307e61bf091cc8677"
server_key_iv: "b999b8530ece4a1072438bd5472966f7"
server_key_version: "encryptionkey@0"
I20260924 07:08:33.086022 19090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:33.098119 19090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:33.101693 19090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:33.101861 19090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:33.102423 19090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:08:33.103223 19090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:08:33.103274 19090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:33.103312 19090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:08:33.103327 19090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:33.112236 19090 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:35729
I20260924 07:08:33.112264 19218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:35729 every 8 connection(s)
I20260924 07:08:33.113174 19090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:08:33.117738 19221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:33.115468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.3:60205)
0924 07:08:33.116512 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:33.116744 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":519,"threads_started":1}
W20260924 07:08:33.118435 19219 heartbeater.cc:644] Failed to heartbeat to 127.3.217.62:42227 (0 consecutive failures): Network error: Failed to ping master at 127.3.217.62:42227: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
I20260924 07:08:33.119313 19221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:33.119010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.3:44143)
0924 07:08:33.119169 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:33.119231 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 07:08:33.119822 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 19090
I20260924 07:08:33.120328 19221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:33.119994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.3:40037)
0924 07:08:33.120111 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:33.120233 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 07:08:33.120612 19219 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 07:08:33.771400 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:33.770942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.1:50425)
0924 07:08:33.771216 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:33.771279 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":74,"threads_started":1}
I20260924 07:08:33.924525 19224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:33.924151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.2:34305)
0924 07:08:33.924376 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:33.924430 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":69,"threads_started":1}
I20260924 07:08:34.121973 19225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:34.121467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.3:40379)
0924 07:08:34.121775 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:34.121834 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":96,"threads_started":1}
I20260924 07:08:34.291556 19226 raft_consensus.cc:482] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:08:34.291887 19226 raft_consensus.cc:504] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:34.293247 19226 leader_election.cc:288] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c837b29bb524c38890cc1328b014f4b (127.3.217.3:35729), 55769f9844444e4a9ce9d7062d8efcf4 (127.3.217.2:35231)
Sep 24 07:08:34 dist-test-slave-2p7j krb5kdc[17643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233712, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.3@KRBTEST.COM
Sep 24 07:08:34 dist-test-slave-2p7j krb5kdc[17643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233712, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.2@KRBTEST.COM
I20260924 07:08:34.316155 19229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:34.293708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:37547 (local address 127.3.217.3:35729)
0924 07:08:34.294153 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:34.294181 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:34.294223 (+ 42us) server_negotiation.cc:407] Connection header received
0924 07:08:34.294924 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:34.294931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:34.295313 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:34.295664 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:34.297107 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:34.298115 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:34.299619 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:34.300175 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:34.300337 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:34.310744 (+ 10407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:34.310754 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:34.310778 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:34.310828 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:34.312955 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:34.313634 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:34.313641 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:34.313649 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:34.313828 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:34.314154 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:34.314157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:34.314158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:34.314874 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:34.315005 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:34.315408 (+ 403us) server_negotiation.cc:299] Negotiation successful
0924 07:08:34.315777 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":77,"threads_started":1}
I20260924 07:08:34.316593 19230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:34.293913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:46367 (local address 127.3.217.2:35231)
0924 07:08:34.294310 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:34.294335 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:34.294378 (+ 43us) server_negotiation.cc:407] Connection header received
0924 07:08:34.295185 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:34.295192 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:34.295631 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:34.295931 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:34.297109 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:34.298192 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:34.299660 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:34.300206 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:34.300394 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:34.309229 (+ 8835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:34.309239 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:34.309269 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:34.309340 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:34.312578 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:34.313316 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:34.313324 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:34.313329 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:34.313481 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:34.313878 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:34.313881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:34.313883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:34.314471 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:34.314608 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:34.316053 (+ 1445us) server_negotiation.cc:299] Negotiation successful
0924 07:08:34.316394 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":101,"threads_started":1}
I20260924 07:08:34.317158 19228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:34.293589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:35231 (local address 127.3.217.1:46367)
0924 07:08:34.294004 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:34.294017 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:34.294095 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:34.295879 (+ 1784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:34.295882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:34.296391 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:34.296936 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:34.296953 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:34.298349 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:34.298352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:34.299460 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:34.299471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:34.300349 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:34.300353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:34.300595 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:34.305795 (+ 5200us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:34.305808 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:34.308680 (+ 2872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:34.312756 (+ 4076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:34.312764 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:34.312785 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:34.313180 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:34.313586 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:34.313590 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:34.313592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:34.313762 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:34.314667 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:34.314674 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:34.315064 (+ 390us) client_negotiation.cc:769] Sending connection context
0924 07:08:34.315277 (+ 213us) client_negotiation.cc:240] Negotiation successful
0924 07:08:34.316959 (+ 1682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":179,"thread_start_us":114,"threads_started":1}
I20260924 07:08:34.317480 19227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:34.293572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:35729 (local address 127.3.217.1:37547)
0924 07:08:34.293861 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:34.293878 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:34.294015 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:34.295771 (+ 1756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:34.295777 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:34.296380 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:34.296934 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:34.296954 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:34.298265 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:34.298268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:34.299459 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:34.299478 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:34.300307 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:34.300310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:34.300596 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:34.305779 (+ 5183us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:34.305802 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:34.310355 (+ 4553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:34.313221 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:34.313225 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:34.313227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:34.313494 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:34.313952 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:34.313955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:34.313958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:34.314051 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:34.314990 (+ 939us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:34.314993 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:34.315103 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 07:08:34.315279 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 07:08:34.317062 (+ 1783us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":1503,"spinlock_wait_cycles":26752,"thread_start_us":68,"threads_started":1}
W20260924 07:08:34.320472 18842 leader_election.cc:341] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c837b29bb524c38890cc1328b014f4b (127.3.217.3:35729): Not found: Tablet not found: 9da85943de164c978ced1f9a6398b3ea
W20260924 07:08:34.320713 18841 leader_election.cc:341] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55769f9844444e4a9ce9d7062d8efcf4 (127.3.217.2:35231): Not found: Tablet not found: 9da85943de164c978ced1f9a6398b3ea
I20260924 07:08:34.320847 18841 leader_election.cc:302] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c915cd7cabc242f9884a2dfc5c20106c; no voters: 55769f9844444e4a9ce9d7062d8efcf4, 8c837b29bb524c38890cc1328b014f4b
I20260924 07:08:34.322494 19226 raft_consensus.cc:2726] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:08:34.772692 19227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:34.772324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.1:44133)
0924 07:08:34.772512 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:34.772583 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 07:08:34.925871 19231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:34.925349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.2:60945)
0924 07:08:34.925672 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:34.925744 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":94,"threads_started":1}
I20260924 07:08:35.123359 19232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:35.122868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.3:59199)
0924 07:08:35.123170 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:35.123237 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":97,"threads_started":1}
I20260924 07:08:35.774245 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:35.773636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.1:43889)
0924 07:08:35.773983 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:35.774106 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":84,"threads_started":1}
I20260924 07:08:35.927352 19234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:35.926795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.2:60785)
0924 07:08:35.927132 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:35.927218 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":78,"threads_started":1}
I20260924 07:08:36.085577 19235 raft_consensus.cc:482] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:08:36.085803 19235 raft_consensus.cc:504] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:36.086341 19235 leader_election.cc:288] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c837b29bb524c38890cc1328b014f4b (127.3.217.3:35729), 55769f9844444e4a9ce9d7062d8efcf4 (127.3.217.2:35231)
W20260924 07:08:36.087248 18841 leader_election.cc:341] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55769f9844444e4a9ce9d7062d8efcf4 (127.3.217.2:35231): Not found: Tablet not found: 9da85943de164c978ced1f9a6398b3ea
W20260924 07:08:36.087422 18842 leader_election.cc:341] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c837b29bb524c38890cc1328b014f4b (127.3.217.3:35729): Not found: Tablet not found: 9da85943de164c978ced1f9a6398b3ea
I20260924 07:08:36.087564 18842 leader_election.cc:302] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c915cd7cabc242f9884a2dfc5c20106c; no voters: 55769f9844444e4a9ce9d7062d8efcf4, 8c837b29bb524c38890cc1328b014f4b
I20260924 07:08:36.087730 19235 raft_consensus.cc:2726] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:08:36.124812 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:36.124265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.3:40031)
0924 07:08:36.124604 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:36.124684 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":95,"threads_started":1}
I20260924 07:08:36.776048 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:36.775362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.1:33393)
0924 07:08:36.775758 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:36.775917 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":151,"threads_started":1}
I20260924 07:08:36.928862 19238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:36.928341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.2:41739)
0924 07:08:36.928692 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:36.928767 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":98,"threads_started":1}
I20260924 07:08:37.126493 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:37.125906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.3:43421)
0924 07:08:37.126288 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:37.126376 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":118,"threads_started":1}
I20260924 07:08:37.777851 19240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:37.777286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.1:35533)
0924 07:08:37.777673 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:37.777747 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":99,"threads_started":1}
I20260924 07:08:37.930308 19241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:37.929700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.2:42669)
0924 07:08:37.930082 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:37.930169 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":82,"threads_started":1}
I20260924 07:08:38.128042 19242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:38.127576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.3:56239)
0924 07:08:38.127890 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:38.127947 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
I20260924 07:08:38.411012 19243 raft_consensus.cc:482] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:08:38.411201 19243 raft_consensus.cc:504] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:38.411747 19243 leader_election.cc:288] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c837b29bb524c38890cc1328b014f4b (127.3.217.3:35729), 55769f9844444e4a9ce9d7062d8efcf4 (127.3.217.2:35231)
W20260924 07:08:38.412843 18841 leader_election.cc:341] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55769f9844444e4a9ce9d7062d8efcf4 (127.3.217.2:35231): Not found: Tablet not found: 9da85943de164c978ced1f9a6398b3ea
W20260924 07:08:38.413002 18842 leader_election.cc:341] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c837b29bb524c38890cc1328b014f4b (127.3.217.3:35729): Not found: Tablet not found: 9da85943de164c978ced1f9a6398b3ea
I20260924 07:08:38.413096 18842 leader_election.cc:302] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c915cd7cabc242f9884a2dfc5c20106c; no voters: 55769f9844444e4a9ce9d7062d8efcf4, 8c837b29bb524c38890cc1328b014f4b
I20260924 07:08:38.413275 19243 raft_consensus.cc:2726] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:08:38.779381 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:38.778821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.1:36955)
0924 07:08:38.779195 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:38.779275 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":128,"threads_started":1}
I20260924 07:08:38.931890 19245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:38.931282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.2:53083)
0924 07:08:38.931658 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:38.931742 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":109,"threads_started":1}
I20260924 07:08:39.129444 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:39.128894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.3:46291)
0924 07:08:39.129235 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:39.129318 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":108,"threads_started":1}
I20260924 07:08:39.780936 19247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:39.780385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.1:37423)
0924 07:08:39.780755 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:39.780833 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":100,"threads_started":1}
I20260924 07:08:39.933465 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:39.932848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.2:33885)
0924 07:08:39.933282 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:39.933351 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":127,"threads_started":1}
I20260924 07:08:40.131047 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:40.130420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.3:54531)
0924 07:08:40.130855 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:40.130922 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":171,"threads_started":1}
I20260924 07:08:40.782294 19250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:40.781870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.1:60221)
0924 07:08:40.782127 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:40.782182 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":84,"threads_started":1}
I20260924 07:08:40.935293 19251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:40.934705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.2:54757)
0924 07:08:40.935107 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:40.935184 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":131,"threads_started":1}
I20260924 07:08:41.005185 19252 raft_consensus.cc:482] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:08:41.005355 19252 raft_consensus.cc:504] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:41.005843 19252 leader_election.cc:288] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c837b29bb524c38890cc1328b014f4b (127.3.217.3:35729), 55769f9844444e4a9ce9d7062d8efcf4 (127.3.217.2:35231)
W20260924 07:08:41.006776 18841 leader_election.cc:341] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55769f9844444e4a9ce9d7062d8efcf4 (127.3.217.2:35231): Not found: Tablet not found: 9da85943de164c978ced1f9a6398b3ea
W20260924 07:08:41.006870 18842 leader_election.cc:341] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c837b29bb524c38890cc1328b014f4b (127.3.217.3:35729): Not found: Tablet not found: 9da85943de164c978ced1f9a6398b3ea
I20260924 07:08:41.006935 18842 leader_election.cc:302] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c915cd7cabc242f9884a2dfc5c20106c; no voters: 55769f9844444e4a9ce9d7062d8efcf4, 8c837b29bb524c38890cc1328b014f4b
I20260924 07:08:41.007194 19252 raft_consensus.cc:2726] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:08:41.132541 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:41.132027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.3:39011)
0924 07:08:41.132369 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:41.132428 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":134,"threads_started":1}
I20260924 07:08:41.783838 19254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:41.783227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.1:39487)
0924 07:08:41.783631 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:41.783731 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":132,"threads_started":1}
I20260924 07:08:41.936875 19255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:41.936349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.2:47995)
0924 07:08:41.936696 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:41.936762 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":127,"threads_started":1}
I20260924 07:08:42.134153 19256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:42.133597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.3:52233)
0924 07:08:42.133972 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:42.134048 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":129,"threads_started":1}
I20260924 07:08:42.785594 19257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:42.785075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.1:38811)
0924 07:08:42.785396 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:42.785466 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":95,"threads_started":1}
I20260924 07:08:42.938349 19258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:42.937833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.2:52805)
0924 07:08:42.938170 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:42.938249 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":100,"threads_started":1}
I20260924 07:08:43.135762 19259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:43.135219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.3:33577)
0924 07:08:43.135590 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:43.135665 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":101,"threads_started":1}
I20260924 07:08:43.409528 19260 raft_consensus.cc:482] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:08:43.409667 19260 raft_consensus.cc:504] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c837b29bb524c38890cc1328b014f4b" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35729 } } peers { permanent_uuid: "c915cd7cabc242f9884a2dfc5c20106c" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 38149 } } peers { permanent_uuid: "55769f9844444e4a9ce9d7062d8efcf4" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 35231 } }
I20260924 07:08:43.410087 19260 leader_election.cc:288] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c837b29bb524c38890cc1328b014f4b (127.3.217.3:35729), 55769f9844444e4a9ce9d7062d8efcf4 (127.3.217.2:35231)
W20260924 07:08:43.411067 18842 leader_election.cc:341] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c837b29bb524c38890cc1328b014f4b (127.3.217.3:35729): Not found: Tablet not found: 9da85943de164c978ced1f9a6398b3ea
W20260924 07:08:43.411194 18841 leader_election.cc:341] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55769f9844444e4a9ce9d7062d8efcf4 (127.3.217.2:35231): Not found: Tablet not found: 9da85943de164c978ced1f9a6398b3ea
I20260924 07:08:43.411348 18841 leader_election.cc:302] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c915cd7cabc242f9884a2dfc5c20106c; no voters: 55769f9844444e4a9ce9d7062d8efcf4, 8c837b29bb524c38890cc1328b014f4b
I20260924 07:08:43.411531 19260 raft_consensus.cc:2726] T 9da85943de164c978ced1f9a6398b3ea P c915cd7cabc242f9884a2dfc5c20106c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:08:43.787040 19261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:43.786489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.1:42591)
0924 07:08:43.786829 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:43.786917 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":95,"threads_started":1}
I20260924 07:08:43.939802 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:43.939170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.2:37505)
0924 07:08:43.939573 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:43.939661 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":91,"threads_started":1}
I20260924 07:08:44.137421 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:44.136828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.3:55047)
0924 07:08:44.137210 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:44.137300 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":136,"threads_started":1}
I20260924 07:08:44.788884 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:44.788179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.1:48103)
0924 07:08:44.788671 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:44.788770 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":121,"threads_started":1}
I20260924 07:08:44.941323 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:44.940748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.2:37755)
0924 07:08:44.941106 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:44.941207 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":107,"threads_started":1}
I20260924 07:08:45.139106 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:45.138517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.3:44759)
0924 07:08:45.138877 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:45.138983 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":113,"threads_started":1}
I20260924 07:08:45.790581 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:45.789942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.1:50985)
0924 07:08:45.790329 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:45.790448 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":108,"threads_started":1}
I20260924 07:08:45.942782 19268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:45.942169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.2:54151)
0924 07:08:45.942555 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:45.942646 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":76,"threads_started":1}
I20260924 07:08:46.140640 19269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:46.140097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.3:54847)
0924 07:08:46.140452 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:46.140521 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":105,"threads_started":1}
I20260924 07:08:46.792054 19270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:46.791482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.1:56421)
0924 07:08:46.791857 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:46.791925 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":98,"threads_started":1}
I20260924 07:08:46.944420 19271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:46.943751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.2:50963)
0924 07:08:46.944131 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:46.944283 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":82,"threads_started":1}
I20260924 07:08:47.142143 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:47.141629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.3:36283)
0924 07:08:47.141940 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:47.142015 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":85,"threads_started":1}
I20260924 07:08:47.793699 19273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:47.793047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.1:49481)
0924 07:08:47.793475 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:47.793563 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":86,"threads_started":1}
I20260924 07:08:47.946053 19274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:47.945629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42227 (local address 127.3.217.2:42953)
0924 07:08:47.945886 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:47.945940 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:42227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260924 07:08:48.130589 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 18789
I20260924 07:08:48.137431 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 18957
I20260924 07:08:48.142961 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 19090
2026-09-24T07:08:48Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19317 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:08:48 dist-test-slave-2p7j krb5kdc[19277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:08:48 dist-test-slave-2p7j krb5kdc[19277](info): set up 2 sockets
Sep 24 07:08:48 dist-test-slave-2p7j krb5kdc[19277](info): commencing operation
krb5kdc: starting...
W20260924 07:08:49.128944 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:08:49 dist-test-slave-2p7j krb5kdc[19277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:08:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:08:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:08:49.260494 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:37863
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:37863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:08:49.369441 19290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:49.369665 19290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:49.369702 19290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:49.373585 19290 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 07:08:49.373632 19290 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 07:08:49.373646 19290 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 07:08:49.373658 19290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:08:49.373673 19290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:49.373689 19290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:49.373703 19290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:08:49.373720 19290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:08:49.378769 19290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:37863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:37863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:49.379702 19290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:49.380610 19290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:49.389905 19298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:49.390009 19290 server_base.cc:1034] running on GCE node
W20260924 07:08:49.389906 19296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:49.390072 19295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:49.391005 19290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:49.392155 19290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:49.395572 19290 hybrid_clock.cc:648] HybridClock initialized: now 1790233729395470 us; error 57 us; skew 500 ppm
Sep 24 07:08:49 dist-test-slave-2p7j krb5kdc[19277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233729, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:49.406344 19290 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:08:49.407619 19290 webserver.cc:460] Webserver started at http://127.3.217.62:36079/ using document root <none> and password file <none>
I20260924 07:08:49.408206 19290 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:49.408286 19290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:08:49.408511 19290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:08:49.410786 19290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "011e0254f5814c418c1210dd0c6a4c93"
format_stamp: "Formatted at 2026-09-24 07:08:49 on dist-test-slave-2p7j"
server_key: "f328d2cae7e9a9a79c21825510c92360"
server_key_iv: "3f7b447fc44f7d9a4b973eeb1da0b67f"
server_key_version: "encryptionkey@0"
I20260924 07:08:49.411919 19290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "011e0254f5814c418c1210dd0c6a4c93"
format_stamp: "Formatted at 2026-09-24 07:08:49 on dist-test-slave-2p7j"
server_key: "f328d2cae7e9a9a79c21825510c92360"
server_key_iv: "3f7b447fc44f7d9a4b973eeb1da0b67f"
server_key_version: "encryptionkey@0"
I20260924 07:08:49.416898 19290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 07:08:49.419643 19305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:49.420976 19290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:08:49.421094 19290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "011e0254f5814c418c1210dd0c6a4c93"
format_stamp: "Formatted at 2026-09-24 07:08:49 on dist-test-slave-2p7j"
server_key: "f328d2cae7e9a9a79c21825510c92360"
server_key_iv: "3f7b447fc44f7d9a4b973eeb1da0b67f"
server_key_version: "encryptionkey@0"
I20260924 07:08:49.421226 19290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:49.456310 19290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:49.459450 19290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:49.459676 19290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:49.467149 19290 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:37863
I20260924 07:08:49.467655 19357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:37863 every 8 connection(s)
I20260924 07:08:49.468220 19290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:08:49.470837 19358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:08:49.476024 19358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93: Bootstrap starting.
I20260924 07:08:49.477312 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 19290
I20260924 07:08:49.477427 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:08:49.477703 3940 external_mini_cluster.cc:1428] Setting key d902f8e0cdc3838db60ba87f3ae3094a
I20260924 07:08:49.478384 19358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93: Neither blocks nor log segments found. Creating new log.
I20260924 07:08:49.479199 19358 log.cc:824] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93: Log is configured to *not* fsync() on all Append() calls
I20260924 07:08:49.481098 19358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93: No bootstrap required, opened a new log
I20260924 07:08:49.484867 19358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011e0254f5814c418c1210dd0c6a4c93" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37863 } }
I20260924 07:08:49.485143 19358 raft_consensus.cc:374] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:08:49.485201 19358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 011e0254f5814c418c1210dd0c6a4c93, State: Initialized, Role: FOLLOWER
I20260924 07:08:49.485802 19358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011e0254f5814c418c1210dd0c6a4c93" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37863 } }
I20260924 07:08:49.486001 19358 raft_consensus.cc:388] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:08:49.486088 19358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:08:49.486227 19358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 07:08:49 dist-test-slave-2p7j krb5kdc[19277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:08:49.487303 19358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011e0254f5814c418c1210dd0c6a4c93" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37863 } }
I20260924 07:08:49.487630 19358 leader_election.cc:302] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 011e0254f5814c418c1210dd0c6a4c93; no voters:
I20260924 07:08:49.487884 19358 leader_election.cc:288] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:08:49.488019 19363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:08:49.488538 19363 raft_consensus.cc:686] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 1 LEADER]: Becoming Leader. State: Replica: 011e0254f5814c418c1210dd0c6a4c93, State: Running, Role: LEADER
I20260924 07:08:49.488885 19363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011e0254f5814c418c1210dd0c6a4c93" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37863 } }
I20260924 07:08:49.488933 19358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:08:49.490653 19364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "011e0254f5814c418c1210dd0c6a4c93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011e0254f5814c418c1210dd0c6a4c93" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37863 } } }
I20260924 07:08:49.490666 19365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 011e0254f5814c418c1210dd0c6a4c93. Latest consensus state: current_term: 1 leader_uuid: "011e0254f5814c418c1210dd0c6a4c93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011e0254f5814c418c1210dd0c6a4c93" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37863 } } }
I20260924 07:08:49.490823 19364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [sys.catalog]: This master's current role is: LEADER
I20260924 07:08:49.490830 19365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [sys.catalog]: This master's current role is: LEADER
I20260924 07:08:49.493186 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:49.478717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.3.217.62:37863)
0924 07:08:49.479087 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:49.479102 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:49.479134 (+ 32us) server_negotiation.cc:407] Connection header received
0924 07:08:49.479798 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:49.479816 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:49.480118 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:49.480365 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:49.481210 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:49.482500 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:49.483780 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:49.484278 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:49.484416 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:49.487330 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:49.487335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:49.487346 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:49.487372 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:49.489360 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:49.489830 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:49.489836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:49.489842 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:49.489975 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:49.490263 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:49.490266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:49.490268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:49.490764 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:49.491098 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:49.491940 (+ 842us) server_negotiation.cc:299] Negotiation successful
0924 07:08:49.492237 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":99,"threads_started":1}
W20260924 07:08:49.494174 19377 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:08:49.494236 19377 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:08:49.495682 19378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:08:49.496383 19378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:08:49.501533 19378 catalog_manager.cc:1347] Generated new cluster ID: de6892e2dc1940018ef0a59b3f8ae3a4
I20260924 07:08:49.501596 19378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:08:49.519467 19378 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:08:49.520963 19378 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:08:49.529690 19378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93: Generated new TSK 0
I20260924 07:08:49.531023 19378 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
I20260924 07:08:49.572443 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:37863
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:08:49.683301 19384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:49.683583 19384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:49.683645 19384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:49.687599 19384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:49.687652 19384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:49.687731 19384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:08:49.693236 19384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:37863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:49.694221 19384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:49.695187 19384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:49.704950 19389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:49.705039 19390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:49.705132 19392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:49.705329 19384 server_base.cc:1034] running on GCE node
I20260924 07:08:49.705695 19384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:49.706279 19384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:49.707461 19384 hybrid_clock.cc:648] HybridClock initialized: now 1790233729707450 us; error 27 us; skew 500 ppm
Sep 24 07:08:49 dist-test-slave-2p7j krb5kdc[19277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233729, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:49.710438 19384 init.cc:377] Logged in from keytab as kudu/127.3.217.1@KRBTEST.COM (short username kudu)
I20260924 07:08:49.711263 19384 webserver.cc:460] Webserver started at http://127.3.217.1:45069/ using document root <none> and password file <none>
I20260924 07:08:49.711812 19384 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:49.711879 19384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:08:49.712077 19384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:08:49.713969 19384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0"
format_stamp: "Formatted at 2026-09-24 07:08:49 on dist-test-slave-2p7j"
server_key: "0e812f3b4e4d176daee294c07b1764c8"
server_key_iv: "cb4db9aa3133272b918111d4418da5db"
server_key_version: "encryptionkey@0"
I20260924 07:08:49.714501 19384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0"
format_stamp: "Formatted at 2026-09-24 07:08:49 on dist-test-slave-2p7j"
server_key: "0e812f3b4e4d176daee294c07b1764c8"
server_key_iv: "cb4db9aa3133272b918111d4418da5db"
server_key_version: "encryptionkey@0"
I20260924 07:08:49.718148 19384 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 07:08:49.720786 19399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:49.722554 19384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:08:49.722664 19384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0"
format_stamp: "Formatted at 2026-09-24 07:08:49 on dist-test-slave-2p7j"
server_key: "0e812f3b4e4d176daee294c07b1764c8"
server_key_iv: "cb4db9aa3133272b918111d4418da5db"
server_key_version: "encryptionkey@0"
I20260924 07:08:49.722765 19384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:49.745016 19384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:49.748960 19384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:49.749193 19384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:49.749927 19384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:08:49.750941 19384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:08:49.750996 19384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:49.751055 19384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:08:49.751084 19384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:49.761440 19384 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:34653
I20260924 07:08:49.761461 19512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:34653 every 8 connection(s)
I20260924 07:08:49.762815 19384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:08:49.769436 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 19384
I20260924 07:08:49.769603 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:08:49.770017 3940 external_mini_cluster.cc:1428] Setting key 24ab051164673d4784c8beea513d4ee2
Sep 24 07:08:49 dist-test-slave-2p7j krb5kdc[19277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233729, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:08:49.779831 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:49.764850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:39711 (local address 127.3.217.62:37863)
0924 07:08:49.764965 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:49.764968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:49.766288 (+ 1320us) server_negotiation.cc:407] Connection header received
0924 07:08:49.767172 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:49.767177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:49.767351 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:49.767440 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:49.768676 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:49.769212 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:49.771202 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:49.771538 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:49.771664 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:49.774975 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:49.774980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:49.774983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:49.775034 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:49.777091 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:49.777847 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:49.777852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:49.777854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:49.777981 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:49.778408 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:49.778412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:49.778414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:49.778706 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:49.778839 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:49.779350 (+ 511us) server_negotiation.cc:299] Negotiation successful
0924 07:08:49.779565 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:08:49.780397 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:49.765154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.1:39711)
0924 07:08:49.766103 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:49.766147 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:49.766884 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:49.767668 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:49.767674 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:49.768042 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:49.768504 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:49.768518 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:49.769447 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:49.769452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:49.771046 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:49.771062 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:49.771736 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:49.771742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:49.771933 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:49.772562 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:49.772581 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:49.774605 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:49.777238 (+ 2633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:49.777248 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:49.777258 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:49.777667 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:49.778138 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:49.778141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:49.778143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:49.778295 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:49.778866 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:49.778871 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:49.779104 (+ 233us) client_negotiation.cc:769] Sending connection context
0924 07:08:49.779272 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 07:08:49.779567 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":560,"threads_started":1}
I20260924 07:08:49.781570 19513 heartbeater.cc:347] Connected to a master server at 127.3.217.62:37863
I20260924 07:08:49.781797 19513 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.217.2@KRBTEST.COM; defaulting to no policy
I20260924 07:08:49.782251 19513 heartbeater.cc:511] Master 127.3.217.62:37863 requested a full tablet report, sending...
I20260924 07:08:49.783829 19321 ts_manager.cc:194] Registered new tserver with Master: dd76bdd5c0bf4b93a9aa8a8022b3ebb0 (127.3.217.1:34653)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.2@KRBTEST.COM" created.
I20260924 07:08:49.785184 19321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.1@KRBTEST.COM'} at 127.3.217.1:39711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
I20260924 07:08:49.803591 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:37863
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:08:49.913095 19518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:49.913343 19518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:49.913399 19518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:49.918082 19518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:49.918215 19518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:49.918345 19518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:08:49.924851 19518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:37863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:49.925863 19518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:49.926849 19518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:08:49.936313 19523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:49.936296 19524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:49.936296 19526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:49.936463 19518 server_base.cc:1034] running on GCE node
I20260924 07:08:49.936915 19518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:49.937474 19518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:49.938733 19518 hybrid_clock.cc:648] HybridClock initialized: now 1790233729938698 us; error 33 us; skew 500 ppm
Sep 24 07:08:49 dist-test-slave-2p7j krb5kdc[19277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233729, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:49.942287 19518 init.cc:377] Logged in from keytab as kudu/127.3.217.2@KRBTEST.COM (short username kudu)
I20260924 07:08:49.943122 19518 webserver.cc:460] Webserver started at http://127.3.217.2:34709/ using document root <none> and password file <none>
I20260924 07:08:49.943699 19518 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:49.943766 19518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:08:49.943954 19518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:08:49.945865 19518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "6088e628e91d476985b2dce6dbc76d2f"
format_stamp: "Formatted at 2026-09-24 07:08:49 on dist-test-slave-2p7j"
server_key: "73073cb316d13edec636e332d72a3462"
server_key_iv: "ecfd326d53ddba8e0d6e598b9c3d2b47"
server_key_version: "encryptionkey@0"
I20260924 07:08:49.946373 19518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "6088e628e91d476985b2dce6dbc76d2f"
format_stamp: "Formatted at 2026-09-24 07:08:49 on dist-test-slave-2p7j"
server_key: "73073cb316d13edec636e332d72a3462"
server_key_iv: "ecfd326d53ddba8e0d6e598b9c3d2b47"
server_key_version: "encryptionkey@0"
I20260924 07:08:49.950174 19518 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:08:49.952719 19533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:49.954414 19518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:08:49.954581 19518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "6088e628e91d476985b2dce6dbc76d2f"
format_stamp: "Formatted at 2026-09-24 07:08:49 on dist-test-slave-2p7j"
server_key: "73073cb316d13edec636e332d72a3462"
server_key_iv: "ecfd326d53ddba8e0d6e598b9c3d2b47"
server_key_version: "encryptionkey@0"
I20260924 07:08:49.954715 19518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:49.966593 19518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:49.969815 19518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:49.970000 19518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:49.970584 19518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:08:49.971382 19518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:08:49.971431 19518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:49.971480 19518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:08:49.971541 19518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:49.980103 19518 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:38221
I20260924 07:08:49.980144 19646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:38221 every 8 connection(s)
I20260924 07:08:49.981349 19518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:08:49.990629 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 19518
I20260924 07:08:49.990756 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:08:49.991065 3940 external_mini_cluster.cc:1428] Setting key 592d16993cfb14f4ec1cc918fd001e48
Sep 24 07:08:49 dist-test-slave-2p7j krb5kdc[19277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233729, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:08:49.995034 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:49.983630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:52183 (local address 127.3.217.62:37863)
0924 07:08:49.983757 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:49.983760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:49.984793 (+ 1033us) server_negotiation.cc:407] Connection header received
0924 07:08:49.985630 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:49.985639 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:49.985686 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:49.985760 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:49.986891 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:49.987406 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:49.988874 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:49.989106 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:49.989205 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:49.991783 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:49.991788 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:49.991791 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:49.991836 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:49.993204 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:49.993661 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:49.993664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:49.993665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:49.993746 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:49.994058 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:49.994061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:49.994062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:49.994220 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:49.994302 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:49.994764 (+ 462us) server_negotiation.cc:299] Negotiation successful
0924 07:08:49.994875 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:08:49.995820 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:49.983972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.2:52183)
0924 07:08:49.984648 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:49.984688 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:49.985422 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:49.985952 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:49.985958 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:49.986307 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:49.986761 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:49.986774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:49.987669 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:49.987674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:49.988744 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:49.988758 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:49.989251 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:49.989255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:49.989424 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:49.989985 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:49.990009 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:49.991647 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:49.993311 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:49.993317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:49.993330 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:49.993563 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:49.993823 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:49.993826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:49.993828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:49.993966 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:49.994314 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:49.994321 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:49.994542 (+ 221us) client_negotiation.cc:769] Sending connection context
0924 07:08:49.994692 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 07:08:49.994989 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":146,"threads_started":1}
I20260924 07:08:49.997121 19647 heartbeater.cc:347] Connected to a master server at 127.3.217.62:37863
I20260924 07:08:49.997340 19647 heartbeater.cc:464] Registering TS with master...
I20260924 07:08:49.997803 19647 heartbeater.cc:511] Master 127.3.217.62:37863 requested a full tablet report, sending...
I20260924 07:08:49.998890 19321 ts_manager.cc:194] Registered new tserver with Master: 6088e628e91d476985b2dce6dbc76d2f (127.3.217.2:38221)
I20260924 07:08:49.999711 19321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.2@KRBTEST.COM'} at 127.3.217.2:52183
WARNING: no policy specified for kudu/127.3.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
I20260924 07:08:50.021011 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:37863
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:08:50.131160 19652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:08:50.131405 19652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:08:50.131451 19652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:08:50.135337 19652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:08:50.135391 19652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:08:50.135448 19652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:08:50.140480 19652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:37863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:08:50.141378 19652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:08:50.142277 19652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:08:50.151409 19652 server_base.cc:1034] running on GCE node
W20260924 07:08:50.151394 19658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:50.151382 19660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:08:50.151613 19657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:08:50.151980 19652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:08:50.152511 19652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:08:50.153687 19652 hybrid_clock.cc:648] HybridClock initialized: now 1790233730153668 us; error 30 us; skew 500 ppm
Sep 24 07:08:50 dist-test-slave-2p7j krb5kdc[19277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233730, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:08:50.156512 19652 init.cc:377] Logged in from keytab as kudu/127.3.217.3@KRBTEST.COM (short username kudu)
I20260924 07:08:50.157311 19652 webserver.cc:460] Webserver started at http://127.3.217.3:41719/ using document root <none> and password file <none>
I20260924 07:08:50.157874 19652 fs_manager.cc:359] Metadata directory not provided
I20260924 07:08:50.157940 19652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:08:50.158133 19652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:08:50.160089 19652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "a2e4050af6ed46daafc8bb828b32d218"
format_stamp: "Formatted at 2026-09-24 07:08:50 on dist-test-slave-2p7j"
server_key: "22c21432ac0b16076fde3a1d97de8393"
server_key_iv: "13355559d3de564e50607a5b788a8647"
server_key_version: "encryptionkey@0"
I20260924 07:08:50.160584 19652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "a2e4050af6ed46daafc8bb828b32d218"
format_stamp: "Formatted at 2026-09-24 07:08:50 on dist-test-slave-2p7j"
server_key: "22c21432ac0b16076fde3a1d97de8393"
server_key_iv: "13355559d3de564e50607a5b788a8647"
server_key_version: "encryptionkey@0"
I20260924 07:08:50.164011 19652 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:08:50.166981 19667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:50.168816 19652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:08:50.168941 19652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "a2e4050af6ed46daafc8bb828b32d218"
format_stamp: "Formatted at 2026-09-24 07:08:50 on dist-test-slave-2p7j"
server_key: "22c21432ac0b16076fde3a1d97de8393"
server_key_iv: "13355559d3de564e50607a5b788a8647"
server_key_version: "encryptionkey@0"
I20260924 07:08:50.169042 19652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:08:50.182117 19652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:08:50.184988 19652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:08:50.185178 19652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:08:50.185735 19652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:08:50.186547 19652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:08:50.186600 19652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:50.186656 19652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:08:50.186690 19652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:08:50.196280 19652 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:41297
I20260924 07:08:50.196302 19780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:41297 every 8 connection(s)
I20260924 07:08:50.197134 19652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:08:50.197646 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 19652
I20260924 07:08:50.197762 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:08:50.198079 3940 external_mini_cluster.cc:1428] Setting key 08e83e1886213c2d45f41037bdf4a9b9
Sep 24 07:08:50 dist-test-slave-2p7j krb5kdc[19277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233730, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:08:50.212323 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:50.199098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:56189 (local address 127.3.217.62:37863)
0924 07:08:50.199248 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:50.199253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:50.200506 (+ 1253us) server_negotiation.cc:407] Connection header received
0924 07:08:50.201374 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:50.201377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:50.201422 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:50.201504 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:50.202578 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.203104 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.204765 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.205115 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.205230 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:50.208083 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:50.208087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:50.208090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:50.208119 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:50.210126 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:50.210715 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:50.210718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:50.210720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:50.210797 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:50.211134 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:50.211136 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:50.211138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:50.211326 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:50.211433 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:50.212008 (+ 575us) server_negotiation.cc:299] Negotiation successful
0924 07:08:50.212172 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:08:50.213192 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:50.199565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.3:56189)
0924 07:08:50.200306 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:50.200358 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:50.201151 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:50.201625 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:50.201631 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:50.201971 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:08:50.202420 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:50.202436 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.203225 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.203228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:50.204599 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:50.204611 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.205261 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.205265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:50.205459 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:50.206130 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:08:50.206153 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:08:50.207907 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:08:50.210268 (+ 2361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:50.210274 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:50.210289 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:50.210575 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:50.210870 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:08:50.210872 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:08:50.210873 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:08:50.211012 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:08:50.211444 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:08:50.211449 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:08:50.211753 (+ 304us) client_negotiation.cc:769] Sending connection context
0924 07:08:50.211967 (+ 214us) client_negotiation.cc:240] Negotiation successful
0924 07:08:50.212308 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":136,"threads_started":1}
I20260924 07:08:50.214391 19781 heartbeater.cc:347] Connected to a master server at 127.3.217.62:37863
I20260924 07:08:50.214641 19781 heartbeater.cc:464] Registering TS with master...
I20260924 07:08:50.215085 19781 heartbeater.cc:511] Master 127.3.217.62:37863 requested a full tablet report, sending...
I20260924 07:08:50.216059 19321 ts_manager.cc:194] Registered new tserver with Master: a2e4050af6ed46daafc8bb828b32d218 (127.3.217.3:41297)
I20260924 07:08:50.216593 19321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.3@KRBTEST.COM'} at 127.3.217.3:56189
I20260924 07:08:50.224231 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:08:50.233721 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:50.226155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41322 (local address 127.3.217.62:37863)
0924 07:08:50.226273 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:50.226277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:50.226494 (+ 217us) server_negotiation.cc:407] Connection header received
0924 07:08:50.226578 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:50.226580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:50.226615 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:50.226681 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:08:50.227807 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.228313 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.229173 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.229380 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.229456 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:50.230917 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:08:50.230920 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:08:50.230922 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:08:50.230941 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:08:50.232198 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:50.232619 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:50.232622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:50.232624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:50.232699 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:08:50.232934 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:08:50.232937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:08:50.232939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:08:50.233156 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:08:50.233275 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:50.233483 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 07:08:50.233585 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:08:50.236930 19321 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:08:50.239738 19321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:08:50.254156 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:50.248754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:34653 (local address 127.0.0.1:52820)
0924 07:08:50.249594 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:50.249606 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:50.249743 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:50.250287 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:50.250289 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:50.250302 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:08:50.250366 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:50.250370 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.251347 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.251349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:50.252598 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:50.252611 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.253634 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.253638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:50.253741 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:50.253780 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 07:08:50.253887 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 07:08:50.253994 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":5,"thread_start_us":89,"threads_started":1}
I20260924 07:08:50.254177 19796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:50.249075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52820 (local address 127.3.217.1:34653)
0924 07:08:50.249441 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:50.249455 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:50.249679 (+ 224us) server_negotiation.cc:407] Connection header received
0924 07:08:50.249886 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:50.249892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:50.250049 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:50.250149 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:08:50.250475 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.251221 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.252736 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.253525 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.253629 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:50.253768 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:50.253924 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 07:08:50.254019 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":103,"threads_started":1}
I20260924 07:08:50.254388 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:50.248395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:38221 (local address 127.0.0.1:56790)
0924 07:08:50.248722 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:50.248770 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:50.248894 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:50.249735 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:50.249741 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:50.249766 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:08:50.249912 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:50.249920 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.251862 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.251865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:50.252976 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:50.252985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.254042 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.254045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:50.254134 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:50.254156 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:08:50.254207 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 07:08:50.254277 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":72,"threads_started":1}
I20260924 07:08:50.254861 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:50.249108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:41297 (local address 127.0.0.1:53952)
0924 07:08:50.249828 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:50.249843 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:50.249940 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:50.250425 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:50.250428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:50.250443 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:08:50.250513 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:50.250519 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.252220 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.252226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:50.253403 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:50.253413 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.254486 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.254489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:50.254583 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:50.254604 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:08:50.254676 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 07:08:50.254747 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260924 07:08:50.255252 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:50.248585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56790 (local address 127.3.217.2:38221)
0924 07:08:50.249204 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:50.249218 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:50.249240 (+ 22us) server_negotiation.cc:407] Connection header received
0924 07:08:50.249319 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:50.249325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:50.249550 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:50.250067 (+ 517us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:08:50.250221 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.251661 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.253126 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.253912 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.254476 (+ 564us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:50.254886 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:50.254937 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 07:08:50.255120 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":131,"threads_started":1}
I20260924 07:08:50.255324 19797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:50.249277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53952 (local address 127.3.217.3:41297)
0924 07:08:50.249683 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:50.249695 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:50.250033 (+ 338us) server_negotiation.cc:407] Connection header received
0924 07:08:50.250111 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:50.250117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:50.250279 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:50.250424 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:08:50.250760 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.252023 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.253532 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.254371 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.254470 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:50.254591 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:50.255080 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 07:08:50.255197 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":126,"threads_started":1}
I20260924 07:08:50.257560 19581 tablet_service.cc:1467] Processing CreateTablet for tablet 039ad70d19a045ca831a1fd31ed05189 (DEFAULT_TABLE table=test-table [id=e2fd63c7af634ee59672ac493ab71f0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:08:50.257545 19447 tablet_service.cc:1467] Processing CreateTablet for tablet 039ad70d19a045ca831a1fd31ed05189 (DEFAULT_TABLE table=test-table [id=e2fd63c7af634ee59672ac493ab71f0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:08:50.257619 19715 tablet_service.cc:1467] Processing CreateTablet for tablet 039ad70d19a045ca831a1fd31ed05189 (DEFAULT_TABLE table=test-table [id=e2fd63c7af634ee59672ac493ab71f0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:08:50.258518 19715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039ad70d19a045ca831a1fd31ed05189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:08:50.258543 19447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039ad70d19a045ca831a1fd31ed05189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:08:50.258550 19581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039ad70d19a045ca831a1fd31ed05189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:08:50.264874 19798 tablet_bootstrap.cc:492] T 039ad70d19a045ca831a1fd31ed05189 P a2e4050af6ed46daafc8bb828b32d218: Bootstrap starting.
I20260924 07:08:50.265479 19799 tablet_bootstrap.cc:492] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Bootstrap starting.
I20260924 07:08:50.265828 19800 tablet_bootstrap.cc:492] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f: Bootstrap starting.
I20260924 07:08:50.268715 19798 tablet_bootstrap.cc:654] T 039ad70d19a045ca831a1fd31ed05189 P a2e4050af6ed46daafc8bb828b32d218: Neither blocks nor log segments found. Creating new log.
I20260924 07:08:50.269102 19800 tablet_bootstrap.cc:654] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f: Neither blocks nor log segments found. Creating new log.
I20260924 07:08:50.269972 19799 tablet_bootstrap.cc:654] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Neither blocks nor log segments found. Creating new log.
I20260924 07:08:50.269968 19798 log.cc:824] T 039ad70d19a045ca831a1fd31ed05189 P a2e4050af6ed46daafc8bb828b32d218: Log is configured to *not* fsync() on all Append() calls
I20260924 07:08:50.269968 19800 log.cc:824] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:08:50.271401 19799 log.cc:824] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Log is configured to *not* fsync() on all Append() calls
I20260924 07:08:50.271937 19798 tablet_bootstrap.cc:492] T 039ad70d19a045ca831a1fd31ed05189 P a2e4050af6ed46daafc8bb828b32d218: No bootstrap required, opened a new log
I20260924 07:08:50.271937 19800 tablet_bootstrap.cc:492] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f: No bootstrap required, opened a new log
I20260924 07:08:50.272126 19798 ts_tablet_manager.cc:1397] T 039ad70d19a045ca831a1fd31ed05189 P a2e4050af6ed46daafc8bb828b32d218: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260924 07:08:50.272126 19800 ts_tablet_manager.cc:1397] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260924 07:08:50.273505 19799 tablet_bootstrap.cc:492] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: No bootstrap required, opened a new log
I20260924 07:08:50.273679 19799 ts_tablet_manager.cc:1397] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260924 07:08:50.276115 19798 raft_consensus.cc:348] T 039ad70d19a045ca831a1fd31ed05189 P a2e4050af6ed46daafc8bb828b32d218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:08:50.276114 19800 raft_consensus.cc:348] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:08:50.276410 19798 raft_consensus.cc:374] T 039ad70d19a045ca831a1fd31ed05189 P a2e4050af6ed46daafc8bb828b32d218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:08:50.276410 19800 raft_consensus.cc:374] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:08:50.276467 19800 raft_consensus.cc:729] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6088e628e91d476985b2dce6dbc76d2f, State: Initialized, Role: FOLLOWER
I20260924 07:08:50.276467 19798 raft_consensus.cc:729] T 039ad70d19a045ca831a1fd31ed05189 P a2e4050af6ed46daafc8bb828b32d218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e4050af6ed46daafc8bb828b32d218, State: Initialized, Role: FOLLOWER
I20260924 07:08:50.276939 19798 consensus_queue.cc:260] T 039ad70d19a045ca831a1fd31ed05189 P a2e4050af6ed46daafc8bb828b32d218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:08:50.276932 19800 consensus_queue.cc:260] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:08:50.277185 19799 raft_consensus.cc:348] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:08:50.277647 19799 raft_consensus.cc:374] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:08:50.277755 19799 raft_consensus.cc:729] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd76bdd5c0bf4b93a9aa8a8022b3ebb0, State: Initialized, Role: FOLLOWER
I20260924 07:08:50.277784 19781 heartbeater.cc:503] Master 127.3.217.62:37863 was elected leader, sending a full tablet report...
I20260924 07:08:50.277957 19798 ts_tablet_manager.cc:1428] T 039ad70d19a045ca831a1fd31ed05189 P a2e4050af6ed46daafc8bb828b32d218: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260924 07:08:50.277945 19800 ts_tablet_manager.cc:1428] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 07:08:50.278306 19647 heartbeater.cc:503] Master 127.3.217.62:37863 was elected leader, sending a full tablet report...
I20260924 07:08:50.278321 19799 consensus_queue.cc:260] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:08:50.279601 19799 ts_tablet_manager.cc:1428] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 07:08:50.280130 19513 heartbeater.cc:503] Master 127.3.217.62:37863 was elected leader, sending a full tablet report...
I20260924 07:08:50.282385 19804 raft_consensus.cc:482] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:08:50.282516 19804 raft_consensus.cc:504] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:08:50.283624 19804 leader_election.cc:288] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e4050af6ed46daafc8bb828b32d218 (127.3.217.3:41297), dd76bdd5c0bf4b93a9aa8a8022b3ebb0 (127.3.217.1:34653)
I20260924 07:08:50.286720 19797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:50.283824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:33727 (local address 127.3.217.3:41297)
0924 07:08:50.283924 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:50.283928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:50.283942 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:08:50.284049 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:50.284052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:50.284095 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:50.284151 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:08:50.284399 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.284898 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.285850 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.286241 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.286327 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:50.286383 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:50.286556 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 07:08:50.286617 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 07:08:50.286720 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:50.283748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:41297 (local address 127.3.217.2:33727)
0924 07:08:50.283847 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:50.283861 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:50.283936 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:50.284185 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:50.284187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:50.284204 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:08:50.284310 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:50.284315 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.284989 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.284992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:50.285742 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:50.285750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.286339 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.286342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:50.286444 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:50.286465 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:08:50.286529 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 07:08:50.286598 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 07:08:50.287166 19807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:50.283768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:34653 (local address 127.3.217.2:60671)
0924 07:08:50.284189 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:50.284202 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:08:50.284291 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:08:50.284486 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:08:50.284488 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:08:50.284503 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:08:50.284608 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:50.284614 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.285280 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.285283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:50.286046 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:08:50.286054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.286636 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.286639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:08:50.286732 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:50.286753 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:08:50.286958 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 07:08:50.287033 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":174,"thread_start_us":88,"threads_started":1}
I20260924 07:08:50.287209 19796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:50.283898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:60671 (local address 127.3.217.1:34653)
0924 07:08:50.284005 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:50.284009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:50.284274 (+ 265us) server_negotiation.cc:407] Connection header received
0924 07:08:50.284368 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:50.284371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:50.284413 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:50.284471 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:08:50.284701 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.285188 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.286157 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.286540 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.286624 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:50.286677 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:50.286993 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 07:08:50.287079 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 07:08:50.287231 19735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039ad70d19a045ca831a1fd31ed05189" candidate_uuid: "6088e628e91d476985b2dce6dbc76d2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e4050af6ed46daafc8bb828b32d218" is_pre_election: true
I20260924 07:08:50.287472 19735 raft_consensus.cc:2454] T 039ad70d19a045ca831a1fd31ed05189 P a2e4050af6ed46daafc8bb828b32d218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6088e628e91d476985b2dce6dbc76d2f in term 0.
I20260924 07:08:50.287858 19467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039ad70d19a045ca831a1fd31ed05189" candidate_uuid: "6088e628e91d476985b2dce6dbc76d2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" is_pre_election: true
I20260924 07:08:50.287935 19537 leader_election.cc:302] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6088e628e91d476985b2dce6dbc76d2f, a2e4050af6ed46daafc8bb828b32d218; no voters:
I20260924 07:08:50.288092 19467 raft_consensus.cc:2454] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6088e628e91d476985b2dce6dbc76d2f in term 0.
I20260924 07:08:50.288197 19804 raft_consensus.cc:2781] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:08:50.288269 19804 raft_consensus.cc:482] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:08:50.288311 19804 raft_consensus.cc:3037] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:08:50.289363 19804 raft_consensus.cc:504] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:08:50.289682 19804 leader_election.cc:288] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f [CANDIDATE]: Term 1 election: Requested vote from peers a2e4050af6ed46daafc8bb828b32d218 (127.3.217.3:41297), dd76bdd5c0bf4b93a9aa8a8022b3ebb0 (127.3.217.1:34653)
I20260924 07:08:50.289973 19735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039ad70d19a045ca831a1fd31ed05189" candidate_uuid: "6088e628e91d476985b2dce6dbc76d2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e4050af6ed46daafc8bb828b32d218"
I20260924 07:08:50.290007 19467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039ad70d19a045ca831a1fd31ed05189" candidate_uuid: "6088e628e91d476985b2dce6dbc76d2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0"
I20260924 07:08:50.290112 19467 raft_consensus.cc:3037] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:08:50.290110 19735 raft_consensus.cc:3037] T 039ad70d19a045ca831a1fd31ed05189 P a2e4050af6ed46daafc8bb828b32d218 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:08:50.291427 19735 raft_consensus.cc:2454] T 039ad70d19a045ca831a1fd31ed05189 P a2e4050af6ed46daafc8bb828b32d218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6088e628e91d476985b2dce6dbc76d2f in term 1.
I20260924 07:08:50.291451 19467 raft_consensus.cc:2454] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6088e628e91d476985b2dce6dbc76d2f in term 1.
I20260924 07:08:50.291790 19534 leader_election.cc:302] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6088e628e91d476985b2dce6dbc76d2f, a2e4050af6ed46daafc8bb828b32d218, dd76bdd5c0bf4b93a9aa8a8022b3ebb0; no voters:
I20260924 07:08:50.292080 19804 raft_consensus.cc:2781] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:08:50.292412 19804 raft_consensus.cc:686] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f [term 1 LEADER]: Becoming Leader. State: Replica: 6088e628e91d476985b2dce6dbc76d2f, State: Running, Role: LEADER
I20260924 07:08:50.292889 19804 consensus_queue.cc:237] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:08:50.296087 19320 catalog_manager.cc:5495] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6088e628e91d476985b2dce6dbc76d2f (127.3.217.2). New cstate: current_term: 1 leader_uuid: "6088e628e91d476985b2dce6dbc76d2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } health_report { overall_health: UNKNOWN } } }
I20260924 07:08:50.320792 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:50.316188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56794 (local address 127.3.217.2:38221)
0924 07:08:50.316372 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 07:08:50.316377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:08:50.316393 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:08:50.316481 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:08:50.316484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:08:50.316562 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:08:50.316654 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:08:50.316945 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.317671 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.318808 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:08:50.319123 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:08:50.319212 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:08:50.319686 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:08:50.320291 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:08:50.320431 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 07:08:50.320529 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 07:08:50.320608 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:08:50.327194 19735 raft_consensus.cc:1261] T 039ad70d19a045ca831a1fd31ed05189 P a2e4050af6ed46daafc8bb828b32d218 [term 1 FOLLOWER]: Refusing update from remote peer 6088e628e91d476985b2dce6dbc76d2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:08:50.327245 19467 raft_consensus.cc:1261] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Refusing update from remote peer 6088e628e91d476985b2dce6dbc76d2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:08:50.328015 19808 consensus_queue.cc:1035] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:08:50.328284 19804 consensus_queue.cc:1035] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:08:50.337127 19813 mvcc.cc:205] Tried to move back new op lower bound from 7332797359412260864 to 7332797359282733056. Current Snapshot: MvccSnapshot[applied={T|T < 7332797359412260864 or (T in {7332797359412260864})}]
I20260924 07:08:50.337757 19811 mvcc.cc:205] Tried to move back new op lower bound from 7332797359412260864 to 7332797359282733056. Current Snapshot: MvccSnapshot[applied={T|T < 7332797359412260864 or (T in {7332797359412260864})}]
I20260924 07:08:50.342173 19812 mvcc.cc:205] Tried to move back new op lower bound from 7332797359412260864 to 7332797359282733056. Current Snapshot: MvccSnapshot[applied={T|T < 7332797359412260864 or (T in {7332797359412260864})}]
I20260924 07:08:50.342932 19322 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:08:50.343096 19322 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:08:50.345373 19322 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93: Sending DeleteTablet for 3 replicas of tablet 039ad70d19a045ca831a1fd31ed05189
I20260924 07:08:50.346094 19715 tablet_service.cc:1514] Processing DeleteTablet for tablet 039ad70d19a045ca831a1fd31ed05189 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:08:50 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:53952
I20260924 07:08:50.346087 19447 tablet_service.cc:1514] Processing DeleteTablet for tablet 039ad70d19a045ca831a1fd31ed05189 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:08:50 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:52820
I20260924 07:08:50.346509 19821 tablet_replica.cc:330] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: stopping tablet replica
I20260924 07:08:50.346509 19822 tablet_replica.cc:330] T 039ad70d19a045ca831a1fd31ed05189 P a2e4050af6ed46daafc8bb828b32d218: stopping tablet replica
I20260924 07:08:50.346525 19581 tablet_service.cc:1514] Processing DeleteTablet for tablet 039ad70d19a045ca831a1fd31ed05189 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:08:50 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:56790
I20260924 07:08:50.346792 19822 raft_consensus.cc:2229] T 039ad70d19a045ca831a1fd31ed05189 P a2e4050af6ed46daafc8bb828b32d218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:08:50.346830 19821 raft_consensus.cc:2229] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:08:50.347048 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 19384
I20260924 07:08:50.347095 19822 raft_consensus.cc:2258] T 039ad70d19a045ca831a1fd31ed05189 P a2e4050af6ed46daafc8bb828b32d218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:08:50.347187 19821 raft_consensus.cc:2258] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:08:50.347754 19823 tablet_replica.cc:330] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f: stopping tablet replica
I20260924 07:08:50.348067 19823 raft_consensus.cc:2229] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:08:50.348416 19823 raft_consensus.cc:2258] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:08:50.348634 19822 ts_tablet_manager.cc:1905] T 039ad70d19a045ca831a1fd31ed05189 P a2e4050af6ed46daafc8bb828b32d218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:08:50.349633 19823 ts_tablet_manager.cc:1905] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:08:50.351521 19822 ts_tablet_manager.cc:1918] T 039ad70d19a045ca831a1fd31ed05189 P a2e4050af6ed46daafc8bb828b32d218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:08:50.351626 19822 log.cc:1196] T 039ad70d19a045ca831a1fd31ed05189 P a2e4050af6ed46daafc8bb828b32d218: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/039ad70d19a045ca831a1fd31ed05189
I20260924 07:08:50.351964 19822 ts_tablet_manager.cc:1939] T 039ad70d19a045ca831a1fd31ed05189 P a2e4050af6ed46daafc8bb828b32d218: Deleting consensus metadata
I20260924 07:08:50.352415 19823 ts_tablet_manager.cc:1918] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:08:50.352571 19823 log.cc:1196] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/039ad70d19a045ca831a1fd31ed05189
I20260924 07:08:50.352907 19823 ts_tablet_manager.cc:1939] T 039ad70d19a045ca831a1fd31ed05189 P 6088e628e91d476985b2dce6dbc76d2f: Deleting consensus metadata
I20260924 07:08:50.353220 19309 catalog_manager.cc:4841] TS a2e4050af6ed46daafc8bb828b32d218 (127.3.217.3:41297): tablet 039ad70d19a045ca831a1fd31ed05189 (table test-table [id=e2fd63c7af634ee59672ac493ab71f0f]) successfully deleted
I20260924 07:08:50.353917 19309 catalog_manager.cc:4841] TS 6088e628e91d476985b2dce6dbc76d2f (127.3.217.2:38221): tablet 039ad70d19a045ca831a1fd31ed05189 (table test-table [id=e2fd63c7af634ee59672ac493ab71f0f]) successfully deleted
W20260924 07:08:50.355289 19306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.217.1:34653 (error 108)
I20260924 07:08:50.356143 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 19518
I20260924 07:08:50.356591 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:08:50.356191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:34653 (local address 127.0.0.1:52836)
0924 07:08:50.356384 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 07:08:50.356482 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.1:34653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260924 07:08:50.357352 19306 catalog_manager.cc:4584] TS dd76bdd5c0bf4b93a9aa8a8022b3ebb0 (127.3.217.1:34653): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 039ad70d19a045ca831a1fd31ed05189: Network error: Client connection negotiation failed: client connection to 127.3.217.1:34653: connect: Connection refused (error 111)
I20260924 07:08:50.361475 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 19652
I20260924 07:08:50.366386 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 19290
I20260924 07:09:10.373842 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:37863
--webserver_interface=127.3.217.62
--webserver_port=36079
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:37863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:10.510277 19825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:10.510643 19825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:10.510691 19825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:10.515225 19825 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 07:09:10.515321 19825 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 07:09:10.515336 19825 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 07:09:10.515352 19825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:09:10.515372 19825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:10.515389 19825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:10.515414 19825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:09:10.515440 19825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:09:10.521729 19825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:37863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:37863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=36079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:10.523267 19825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:10.524652 19825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:10.540249 19834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:10.540249 19832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:10.540283 19831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:10.540733 19825 server_base.cc:1034] running on GCE node
I20260924 07:09:10.541571 19825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:10.543229 19825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:10.544414 19825 hybrid_clock.cc:648] HybridClock initialized: now 1790233750544384 us; error 57 us; skew 500 ppm
Sep 24 07:09:10 dist-test-slave-2p7j krb5kdc[19277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233750, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:10.553063 19825 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:09:10.554490 19825 webserver.cc:460] Webserver started at http://127.3.217.62:36079/ using document root <none> and password file <none>
I20260924 07:09:10.555248 19825 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:10.555361 19825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:10.560636 19825 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260924 07:09:10.563807 19841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:10.565680 19825 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 07:09:10.565896 19825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "011e0254f5814c418c1210dd0c6a4c93"
format_stamp: "Formatted at 2026-09-24 07:08:49 on dist-test-slave-2p7j"
server_key: "f328d2cae7e9a9a79c21825510c92360"
server_key_iv: "3f7b447fc44f7d9a4b973eeb1da0b67f"
server_key_version: "encryptionkey@0"
I20260924 07:09:10.566789 19825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:10.602952 19825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:10.607129 19825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:10.607385 19825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:10.616117 19825 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:37863
I20260924 07:09:10.616421 19893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:37863 every 8 connection(s)
I20260924 07:09:10.617513 19825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:09:10.622387 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 19825
I20260924 07:09:10.622531 19894 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:09:10.622864 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:34653
--local_ip_for_outbound_sockets=127.3.217.1
--tserver_master_addrs=127.3.217.62:37863
--webserver_port=45069
--webserver_interface=127.3.217.1
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:09:10.624655 19894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93: Bootstrap starting.
I20260924 07:09:10.641350 19894 log.cc:824] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:10.650058 19894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:09:10.650439 19894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93: Bootstrap complete.
I20260924 07:09:10.654367 19894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011e0254f5814c418c1210dd0c6a4c93" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37863 } }
I20260924 07:09:10.654775 19894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 011e0254f5814c418c1210dd0c6a4c93, State: Initialized, Role: FOLLOWER
I20260924 07:09:10.655246 19894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011e0254f5814c418c1210dd0c6a4c93" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37863 } }
I20260924 07:09:10.655368 19894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:09:10.655440 19894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:09:10.655570 19894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:09:10.657500 19894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011e0254f5814c418c1210dd0c6a4c93" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37863 } }
I20260924 07:09:10.657833 19894 leader_election.cc:302] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 011e0254f5814c418c1210dd0c6a4c93; no voters:
I20260924 07:09:10.658121 19894 leader_election.cc:288] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 07:09:10.658269 19898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 07:09:10.658483 19898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 2 LEADER]: Becoming Leader. State: Replica: 011e0254f5814c418c1210dd0c6a4c93, State: Running, Role: LEADER
I20260924 07:09:10.658864 19898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011e0254f5814c418c1210dd0c6a4c93" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37863 } }
I20260924 07:09:10.659391 19894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:09:10.659834 19898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 011e0254f5814c418c1210dd0c6a4c93. Latest consensus state: current_term: 2 leader_uuid: "011e0254f5814c418c1210dd0c6a4c93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011e0254f5814c418c1210dd0c6a4c93" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37863 } } }
I20260924 07:09:10.660600 19898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:10.660483 19899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "011e0254f5814c418c1210dd0c6a4c93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011e0254f5814c418c1210dd0c6a4c93" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37863 } } }
I20260924 07:09:10.661310 19899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:10.665238 19914 catalog_manager.cc:1259] Loaded cluster ID: de6892e2dc1940018ef0a59b3f8ae3a4
I20260924 07:09:10.665349 19914 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93: loading cluster ID for follower catalog manager: success
I20260924 07:09:10.668229 19914 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93: acquiring CA information for follower catalog manager: success
I20260924 07:09:10.669970 19914 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:09:10.670122 19907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:09:10.673602 19907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:09:10.674175 19907 catalog_manager.cc:1259] Loaded cluster ID: de6892e2dc1940018ef0a59b3f8ae3a4
I20260924 07:09:10.674237 19907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:09:10.675135 19907 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:09:10.675773 19907 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93: Loaded TSK: 0
I20260924 07:09:10.693051 19907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93: Generated new TSK 1
I20260924 07:09:10.693421 19907 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:09:10.762993 19896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:10.763350 19896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:10.763401 19896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:10.768010 19896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:10.768108 19896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:10.768181 19896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:09:10.774328 19896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:34653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=45069
--tserver_master_addrs=127.3.217.62:37863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:10.775759 19896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:10.777063 19896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:10.789031 19922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:10.789003 19921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:10.789135 19924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:10.789605 19896 server_base.cc:1034] running on GCE node
I20260924 07:09:10.790160 19896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:10.791023 19896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:10.792240 19896 hybrid_clock.cc:648] HybridClock initialized: now 1790233750792223 us; error 33 us; skew 500 ppm
Sep 24 07:09:10 dist-test-slave-2p7j krb5kdc[19277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233750, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:10.795785 19896 init.cc:377] Logged in from keytab as kudu/127.3.217.1@KRBTEST.COM (short username kudu)
I20260924 07:09:10.796900 19896 webserver.cc:460] Webserver started at http://127.3.217.1:45069/ using document root <none> and password file <none>
I20260924 07:09:10.797559 19896 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:10.797644 19896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:10.802502 19896 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:09:10.805317 19931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:10.806974 19896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 07:09:10.807134 19896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0"
format_stamp: "Formatted at 2026-09-24 07:08:49 on dist-test-slave-2p7j"
server_key: "0e812f3b4e4d176daee294c07b1764c8"
server_key_iv: "cb4db9aa3133272b918111d4418da5db"
server_key_version: "encryptionkey@0"
I20260924 07:09:10.807886 19896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:10.822129 19896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:10.826079 19896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:10.826303 19896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:10.827024 19896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:10.828645 19938 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 07:09:10.833060 19896 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 07:09:10.833160 19896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260924 07:09:10.833245 19896 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 07:09:10.835529 19896 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 07:09:10.835597 19896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.003s sys 0.000s
I20260924 07:09:10.835979 19938 tablet_bootstrap.cc:492] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Bootstrap starting.
I20260924 07:09:10.846899 19896 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:34653
I20260924 07:09:10.846994 20045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:34653 every 8 connection(s)
I20260924 07:09:10.848126 19896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:09:10.850992 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 19896
I20260924 07:09:10.851538 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:38221
--local_ip_for_outbound_sockets=127.3.217.2
--tserver_master_addrs=127.3.217.62:37863
--webserver_port=34709
--webserver_interface=127.3.217.2
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:09:10.854110 19938 log.cc:824] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Log is configured to *not* fsync() on all Append() calls
Sep 24 07:09:10 dist-test-slave-2p7j krb5kdc[19277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233750, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:09:10.864068 19938 tablet_bootstrap.cc:492] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:09:10.864544 19938 tablet_bootstrap.cc:492] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Bootstrap complete.
I20260924 07:09:10.865679 19938 ts_tablet_manager.cc:1397] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Time spent bootstrapping tablet: real 0.030s user 0.023s sys 0.005s
I20260924 07:09:10.869032 19938 raft_consensus.cc:348] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:09:10.869640 19938 raft_consensus.cc:729] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd76bdd5c0bf4b93a9aa8a8022b3ebb0, State: Initialized, Role: FOLLOWER
I20260924 07:09:10.869728 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:10.850604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:45961 (local address 127.3.217.62:37863)
0924 07:09:10.851219 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:10.851243 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:10.851779 (+ 536us) server_negotiation.cc:407] Connection header received
0924 07:09:10.853177 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:10.853192 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:10.853631 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:10.854030 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:10.855644 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:10.857208 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:10.859031 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:10.859847 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:10.860029 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:10.863588 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:10.863597 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:10.863616 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:10.863657 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:10.865801 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:10.866523 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:10.866529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:10.866535 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:10.866696 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:10.867108 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:10.867111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:10.867113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:10.867615 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:10.867749 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:10.868462 (+ 713us) server_negotiation.cc:299] Negotiation successful
0924 07:09:10.868785 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":124,"threads_started":1}
I20260924 07:09:10.870225 19938 consensus_queue.cc:260] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:09:10.870369 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:10.850444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.1:45961)
0924 07:09:10.851576 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:10.851628 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:10.852675 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:10.854183 (+ 1508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:10.854195 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:10.854684 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:10.855382 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:10.855404 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:10.857417 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:10.857421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:10.858867 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:10.858880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:10.860112 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:10.860116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:10.860396 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:10.861159 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:10.861180 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:10.863292 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:10.865970 (+ 2678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:10.865977 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:10.865994 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:10.866394 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:10.866813 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:10.866816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:10.866818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:10.867009 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:10.867736 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:10.867744 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:10.868109 (+ 365us) client_negotiation.cc:769] Sending connection context
0924 07:09:10.868333 (+ 224us) client_negotiation.cc:240] Negotiation successful
0924 07:09:10.868748 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"thread_start_us":175,"threads_started":1}
I20260924 07:09:10.871347 19938 ts_tablet_manager.cc:1428] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 07:09:10.873034 20046 heartbeater.cc:347] Connected to a master server at 127.3.217.62:37863
I20260924 07:09:10.873307 20046 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:10.873880 20046 heartbeater.cc:511] Master 127.3.217.62:37863 requested a full tablet report, sending...
I20260924 07:09:10.877022 19858 ts_manager.cc:194] Registered new tserver with Master: dd76bdd5c0bf4b93a9aa8a8022b3ebb0 (127.3.217.1:34653)
I20260924 07:09:10.881800 19858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.1@KRBTEST.COM'} at 127.3.217.1:45961
I20260924 07:09:10.885797 20046 heartbeater.cc:503] Master 127.3.217.62:37863 was elected leader, sending a full tablet report...
Sep 24 07:09:10 dist-test-slave-2p7j krb5kdc[19277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233750, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for kudu/127.3.217.1@KRBTEST.COM
I20260924 07:09:10.892921 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:10.880630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:34653 (local address 127.0.0.1:51798)
0924 07:09:10.881428 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:10.881480 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:10.881669 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:10.882264 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:10.882272 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:10.882696 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:10.882856 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:10.882866 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:10.884683 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:10.884687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:10.885831 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:10.885846 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:10.886722 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:10.886727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:10.886873 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:10.887794 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:10.887812 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:10.889565 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:10.891262 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:10.891271 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:10.891277 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:10.891492 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:10.891790 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:10.891793 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:10.891795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:10.891872 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:10.892291 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:10.892296 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:10.892415 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 07:09:10.892536 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 07:09:10.892745 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":137,"threads_started":1}
I20260924 07:09:10.892927 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:10.880777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51798 (local address 127.3.217.1:34653)
0924 07:09:10.881177 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:10.881191 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:10.881628 (+ 437us) server_negotiation.cc:407] Connection header received
0924 07:09:10.881845 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:10.881851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:10.882063 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:10.883011 (+ 948us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:10.883154 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:10.884471 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:10.886004 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:10.886423 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:10.886536 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:10.889706 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:10.889715 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:10.889727 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:10.889749 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:10.891156 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:10.891622 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:10.891628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:10.891629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:10.891705 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:10.891948 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:10.891951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:10.891953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:10.892184 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:10.892297 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:10.892628 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 07:09:10.892781 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":171,"threads_started":1}
I20260924 07:09:10.893903 19980 tablet_service.cc:1514] Processing DeleteTablet for tablet 039ad70d19a045ca831a1fd31ed05189 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:08:50 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:51798
I20260924 07:09:10.894778 20055 tablet_replica.cc:330] stopping tablet replica
I20260924 07:09:10.895092 20055 raft_consensus.cc:2229] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:10.895360 20055 raft_consensus.cc:2258] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:10.896618 20055 ts_tablet_manager.cc:1905] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:10.901068 20055 ts_tablet_manager.cc:1918] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:09:10.901177 20055 log.cc:1196] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/wal/wals/039ad70d19a045ca831a1fd31ed05189
I20260924 07:09:10.901427 20055 ts_tablet_manager.cc:1939] T 039ad70d19a045ca831a1fd31ed05189 P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Deleting consensus metadata
I20260924 07:09:10.903394 19842 catalog_manager.cc:4841] TS dd76bdd5c0bf4b93a9aa8a8022b3ebb0 (127.3.217.1:34653): tablet 039ad70d19a045ca831a1fd31ed05189 (table test-table [id=e2fd63c7af634ee59672ac493ab71f0f]) successfully deleted
W20260924 07:09:10.980036 20050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:10.980278 20050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:10.980320 20050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:10.984361 20050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:10.984433 20050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:10.984490 20050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:09:10.989876 20050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:38221
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=34709
--tserver_master_addrs=127.3.217.62:37863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:10.990974 20050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:10.992128 20050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:11.001657 20061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:11.001700 20060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:11.001701 20063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:11.002014 20050 server_base.cc:1034] running on GCE node
I20260924 07:09:11.002524 20050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:11.003124 20050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:11.004421 20050 hybrid_clock.cc:648] HybridClock initialized: now 1790233751004386 us; error 46 us; skew 500 ppm
Sep 24 07:09:11 dist-test-slave-2p7j krb5kdc[19277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233751, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:11.007094 20050 init.cc:377] Logged in from keytab as kudu/127.3.217.2@KRBTEST.COM (short username kudu)
I20260924 07:09:11.008064 20050 webserver.cc:460] Webserver started at http://127.3.217.2:34709/ using document root <none> and password file <none>
I20260924 07:09:11.008608 20050 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:11.008690 20050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:11.012374 20050 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:09:11.014357 20070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:11.015694 20050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:09:11.015799 20050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "6088e628e91d476985b2dce6dbc76d2f"
format_stamp: "Formatted at 2026-09-24 07:08:49 on dist-test-slave-2p7j"
server_key: "73073cb316d13edec636e332d72a3462"
server_key_iv: "ecfd326d53ddba8e0d6e598b9c3d2b47"
server_key_version: "encryptionkey@0"
I20260924 07:09:11.016331 20050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:11.039126 20050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:11.042840 20050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:11.043030 20050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:11.044095 20050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:11.045403 20050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:11.045477 20050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:11.045538 20050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:11.045568 20050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:11.054128 20050 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:38221
I20260924 07:09:11.054361 20183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:38221 every 8 connection(s)
I20260924 07:09:11.055111 20050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:09:11.060084 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 20050
I20260924 07:09:11.060839 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:41297
--local_ip_for_outbound_sockets=127.3.217.3
--tserver_master_addrs=127.3.217.62:37863
--webserver_port=41719
--webserver_interface=127.3.217.3
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:09:11 dist-test-slave-2p7j krb5kdc[19277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233751, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:09:11.070367 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:11.057202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:54801 (local address 127.3.217.62:37863)
0924 07:09:11.057369 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:11.057373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:11.058083 (+ 710us) server_negotiation.cc:407] Connection header received
0924 07:09:11.059553 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:11.059558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:11.059652 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:11.059753 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:11.061301 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.062018 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.063327 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.063699 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.063818 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:11.066307 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:11.066311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:11.066314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:11.066343 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:11.068156 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:11.068645 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:11.068649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:11.068651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:11.068737 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:11.069141 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:11.069145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:11.069147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:11.069375 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:11.069470 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:11.070058 (+ 588us) server_negotiation.cc:299] Negotiation successful
0924 07:09:11.070195 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 07:09:11.071115 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:11.057448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.2:54801)
0924 07:09:11.057910 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:11.057950 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:11.059271 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:11.059945 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:11.059952 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:11.060346 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:11.061157 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:11.061174 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.062224 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.062227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:11.063201 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:11.063211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.063883 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.063886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:11.064032 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:11.064623 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:11.064639 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:11.066124 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:11.068283 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:11.068288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:11.068300 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:11.068540 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:11.068894 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:11.068897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:11.068899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:11.069048 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:11.069521 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:11.069527 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:11.069765 (+ 238us) client_negotiation.cc:769] Sending connection context
0924 07:09:11.069993 (+ 228us) client_negotiation.cc:240] Negotiation successful
0924 07:09:11.070269 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":104,"threads_started":1}
I20260924 07:09:11.072461 20184 heartbeater.cc:347] Connected to a master server at 127.3.217.62:37863
I20260924 07:09:11.072675 20184 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:11.073143 20184 heartbeater.cc:511] Master 127.3.217.62:37863 requested a full tablet report, sending...
I20260924 07:09:11.074155 19858 ts_manager.cc:194] Registered new tserver with Master: 6088e628e91d476985b2dce6dbc76d2f (127.3.217.2:38221)
I20260924 07:09:11.074693 19858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.2@KRBTEST.COM'} at 127.3.217.2:54801
W20260924 07:09:11.181262 20187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:11.181522 20187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:11.181582 20187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:11.187999 20187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:11.188071 20187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:11.188158 20187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:09:11.193807 20187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:41297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=41719
--tserver_master_addrs=127.3.217.62:37863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:11.194856 20187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:11.195833 20187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:11.206446 20195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:11.206596 20193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:11.206437 20192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:11.206835 20187 server_base.cc:1034] running on GCE node
I20260924 07:09:11.207329 20187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:11.208040 20187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:11.209225 20187 hybrid_clock.cc:648] HybridClock initialized: now 1790233751209212 us; error 42 us; skew 500 ppm
Sep 24 07:09:11 dist-test-slave-2p7j krb5kdc[19277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233751, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:11.212015 20187 init.cc:377] Logged in from keytab as kudu/127.3.217.3@KRBTEST.COM (short username kudu)
I20260924 07:09:11.212815 20187 webserver.cc:460] Webserver started at http://127.3.217.3:41719/ using document root <none> and password file <none>
I20260924 07:09:11.213335 20187 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:11.213407 20187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:11.216938 20187 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:09:11.218909 20202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:11.220538 20187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:09:11.220650 20187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "a2e4050af6ed46daafc8bb828b32d218"
format_stamp: "Formatted at 2026-09-24 07:08:50 on dist-test-slave-2p7j"
server_key: "22c21432ac0b16076fde3a1d97de8393"
server_key_iv: "13355559d3de564e50607a5b788a8647"
server_key_version: "encryptionkey@0"
I20260924 07:09:11.221189 20187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:11.250878 20187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:11.253844 20187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:11.254000 20187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:11.254573 20187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:11.255357 20187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:11.255401 20187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:11.255446 20187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:11.255473 20187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:11.266170 20187 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:41297
I20260924 07:09:11.266232 20315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:41297 every 8 connection(s)
I20260924 07:09:11.267213 20187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:09:11.268463 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 20187
Sep 24 07:09:11 dist-test-slave-2p7j krb5kdc[19277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233751, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:09:11.279467 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:11.269386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57596 (local address 127.3.217.62:37863)
0924 07:09:11.269591 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:11.269596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:11.269808 (+ 212us) server_negotiation.cc:407] Connection header received
0924 07:09:11.270046 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:11.270049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:11.270116 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:11.270198 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:11.271296 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.272118 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.273294 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.273608 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.273711 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:11.275602 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:11.275606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:11.275609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:11.275635 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:11.277486 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:11.277929 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:11.277933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:11.277935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:11.278007 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:11.278468 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:11.278472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:11.278474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:11.278676 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:11.278891 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:11.279228 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 07:09:11.279336 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260924 07:09:11.282938 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:11.269682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:54773 (local address 127.3.217.62:37863)
0924 07:09:11.269916 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:11.269919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:11.271693 (+ 1774us) server_negotiation.cc:407] Connection header received
0924 07:09:11.272737 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:11.272740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:11.272803 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:11.272883 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:11.274222 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.274851 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.276100 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.276323 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.276404 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:11.279028 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:11.279031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:11.279033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:11.279052 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:11.280653 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:11.281117 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:11.281120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:11.281121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:11.281190 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:11.281585 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:11.281590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:11.281591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:11.282012 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:11.282103 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:11.282640 (+ 537us) server_negotiation.cc:299] Negotiation successful
0924 07:09:11.282774 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":72,"threads_started":1}
I20260924 07:09:11.284163 20320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:11.270458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.3:54773)
0924 07:09:11.271414 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:11.271475 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:11.272528 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:11.273090 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:11.273098 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:11.273552 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:11.274097 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:11.274110 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.274968 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.274970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:11.275998 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:11.276007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.276457 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.276461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:11.276625 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:11.277362 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:11.277379 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:11.278879 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:11.280772 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:11.280778 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:11.280793 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:11.281027 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:11.281283 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:11.281286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:11.281288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:11.281418 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:11.282140 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:11.282145 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:11.282380 (+ 235us) client_negotiation.cc:769] Sending connection context
0924 07:09:11.282563 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 07:09:11.282838 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":210,"threads_started":1}
I20260924 07:09:11.285651 20316 heartbeater.cc:347] Connected to a master server at 127.3.217.62:37863
I20260924 07:09:11.285887 20316 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:11.286360 20316 heartbeater.cc:511] Master 127.3.217.62:37863 requested a full tablet report, sending...
I20260924 07:09:11.287357 19858 ts_manager.cc:194] Registered new tserver with Master: a2e4050af6ed46daafc8bb828b32d218 (127.3.217.3:41297)
I20260924 07:09:11.287922 19858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.3@KRBTEST.COM'} at 127.3.217.3:54773
I20260924 07:09:11.293006 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:09:11.297585 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:11.293722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57608 (local address 127.3.217.62:37863)
0924 07:09:11.293852 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:11.293855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:11.294065 (+ 210us) server_negotiation.cc:407] Connection header received
0924 07:09:11.294177 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:11.294180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:11.294266 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:11.294343 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:09:11.294612 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.295122 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.296287 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.296511 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.296598 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:11.296911 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:09:11.297034 (+ 123us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 07:09:11.297465 (+ 431us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57608: token expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 07:09:11.297643 20319 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57608: token expired
W20260924 07:09:11.297770 19787 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.217.62:37863)
I20260924 07:09:11.307003 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:11.298172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.3.217.62:37863)
0924 07:09:11.298302 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:11.298308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:11.298325 (+ 17us) server_negotiation.cc:407] Connection header received
0924 07:09:11.298398 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:11.298401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:11.298446 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:11.298539 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:11.299215 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.299922 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.301218 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.301536 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.301671 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:11.303551 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:11.303556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:11.303559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:11.303585 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:11.305204 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:11.305632 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:11.305636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:11.305638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:11.305733 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:11.306203 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:11.306207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:11.306208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:11.306403 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:11.306535 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:11.306760 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 07:09:11.306852 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 07:09:11.309425 19858 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:09:11.312654 19858 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:09:11.320189 19980 tablet_service.cc:1467] Processing CreateTablet for tablet cf1afd2b5ad1476489d3db5e5271343e (DEFAULT_TABLE table=test-table [id=78a3edee33004a0eb0db1e5b246f9e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:11.320834 19980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf1afd2b5ad1476489d3db5e5271343e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:11.323336 19938 tablet_bootstrap.cc:492] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Bootstrap starting.
I20260924 07:09:11.324215 19938 tablet_bootstrap.cc:654] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:11.324918 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:11.319174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:41297 (local address 127.0.0.1:33420)
0924 07:09:11.319291 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:11.319306 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:11.319386 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:11.320422 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:11.320426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:11.320442 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:11.320549 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:11.320556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.322060 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.322063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:11.323263 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:11.323274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.324451 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.324455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:11.324661 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:11.324678 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 07:09:11.324717 (+ 39us) client_negotiation.cc:240] Negotiation successful
0924 07:09:11.324777 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":112}
I20260924 07:09:11.325031 20326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:11.319362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33420 (local address 127.3.217.3:41297)
0924 07:09:11.319922 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:11.319934 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:11.319956 (+ 22us) server_negotiation.cc:407] Connection header received
0924 07:09:11.320033 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:11.320039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:11.320265 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:11.320414 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:11.320673 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.321937 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.323547 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.324340 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.324445 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:11.324692 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:11.324801 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 07:09:11.324893 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":168,"threads_started":1}
I20260924 07:09:11.325284 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:11.319821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52928 (local address 127.3.217.2:38221)
0924 07:09:11.320132 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:11.320141 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:11.320158 (+ 17us) server_negotiation.cc:407] Connection header received
0924 07:09:11.320218 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:11.320223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:11.320377 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:11.320724 (+ 347us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:11.321052 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.322179 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.323239 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.324319 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.324856 (+ 537us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:11.324985 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:11.325035 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 07:09:11.325135 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":115,"threads_started":1}
I20260924 07:09:11.325569 20325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:11.319398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:38221 (local address 127.0.0.1:52928)
0924 07:09:11.319695 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:11.319714 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:11.319794 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:11.320508 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:11.320510 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:11.320525 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:11.320602 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:11.320608 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.322273 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.322277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:11.323114 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:11.323123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.324451 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.324455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:11.324601 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:11.324650 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 07:09:11.324703 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 07:09:11.324776 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":25,"thread_start_us":114,"threads_started":1}
I20260924 07:09:11.326666 19938 tablet_bootstrap.cc:492] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: No bootstrap required, opened a new log
I20260924 07:09:11.326762 19938 ts_tablet_manager.cc:1397] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:09:11.327215 19938 raft_consensus.cc:348] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:09:11.327338 19938 raft_consensus.cc:374] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:11.327384 19938 raft_consensus.cc:729] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd76bdd5c0bf4b93a9aa8a8022b3ebb0, State: Initialized, Role: FOLLOWER
I20260924 07:09:11.327524 19938 consensus_queue.cc:260] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:09:11.327785 19938 ts_tablet_manager.cc:1428] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:09:11.327816 20250 tablet_service.cc:1467] Processing CreateTablet for tablet cf1afd2b5ad1476489d3db5e5271343e (DEFAULT_TABLE table=test-table [id=78a3edee33004a0eb0db1e5b246f9e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:11.328218 20118 tablet_service.cc:1467] Processing CreateTablet for tablet cf1afd2b5ad1476489d3db5e5271343e (DEFAULT_TABLE table=test-table [id=78a3edee33004a0eb0db1e5b246f9e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:11.328739 20250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf1afd2b5ad1476489d3db5e5271343e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:11.328910 20118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf1afd2b5ad1476489d3db5e5271343e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:11.335057 20329 tablet_bootstrap.cc:492] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218: Bootstrap starting.
I20260924 07:09:11.335171 20330 tablet_bootstrap.cc:492] T cf1afd2b5ad1476489d3db5e5271343e P 6088e628e91d476985b2dce6dbc76d2f: Bootstrap starting.
I20260924 07:09:11.337337 20329 tablet_bootstrap.cc:654] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:11.338039 20329 log.cc:824] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:11.338611 20330 tablet_bootstrap.cc:654] T cf1afd2b5ad1476489d3db5e5271343e P 6088e628e91d476985b2dce6dbc76d2f: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:11.339387 20330 log.cc:824] T cf1afd2b5ad1476489d3db5e5271343e P 6088e628e91d476985b2dce6dbc76d2f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:11.339732 20329 tablet_bootstrap.cc:492] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218: No bootstrap required, opened a new log
I20260924 07:09:11.339926 20329 ts_tablet_manager.cc:1397] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:09:11.341815 20330 tablet_bootstrap.cc:492] T cf1afd2b5ad1476489d3db5e5271343e P 6088e628e91d476985b2dce6dbc76d2f: No bootstrap required, opened a new log
I20260924 07:09:11.342059 20330 ts_tablet_manager.cc:1397] T cf1afd2b5ad1476489d3db5e5271343e P 6088e628e91d476985b2dce6dbc76d2f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 07:09:11.343042 20329 raft_consensus.cc:348] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:09:11.343266 20329 raft_consensus.cc:374] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:11.343298 20329 raft_consensus.cc:729] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e4050af6ed46daafc8bb828b32d218, State: Initialized, Role: FOLLOWER
I20260924 07:09:11.343724 20329 consensus_queue.cc:260] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:09:11.344414 20316 heartbeater.cc:503] Master 127.3.217.62:37863 was elected leader, sending a full tablet report...
I20260924 07:09:11.344604 20329 ts_tablet_manager.cc:1428] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:09:11.345991 20330 raft_consensus.cc:348] T cf1afd2b5ad1476489d3db5e5271343e P 6088e628e91d476985b2dce6dbc76d2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:09:11.346230 20330 raft_consensus.cc:374] T cf1afd2b5ad1476489d3db5e5271343e P 6088e628e91d476985b2dce6dbc76d2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:11.346288 20330 raft_consensus.cc:729] T cf1afd2b5ad1476489d3db5e5271343e P 6088e628e91d476985b2dce6dbc76d2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6088e628e91d476985b2dce6dbc76d2f, State: Initialized, Role: FOLLOWER
I20260924 07:09:11.346726 20330 consensus_queue.cc:260] T cf1afd2b5ad1476489d3db5e5271343e P 6088e628e91d476985b2dce6dbc76d2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:09:11.347311 20184 heartbeater.cc:503] Master 127.3.217.62:37863 was elected leader, sending a full tablet report...
I20260924 07:09:11.347687 20330 ts_tablet_manager.cc:1428] T cf1afd2b5ad1476489d3db5e5271343e P 6088e628e91d476985b2dce6dbc76d2f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260924 07:09:11.349434 20047 tablet.cc:2369] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:09:11.510910 20333 raft_consensus.cc:482] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:09:11.511152 20333 raft_consensus.cc:504] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:09:11.512480 20333 leader_election.cc:288] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6088e628e91d476985b2dce6dbc76d2f (127.3.217.2:38221), dd76bdd5c0bf4b93a9aa8a8022b3ebb0 (127.3.217.1:34653)
I20260924 07:09:11.518265 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:11.513106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:52737 (local address 127.3.217.2:38221)
0924 07:09:11.513261 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:11.513266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:11.513606 (+ 340us) server_negotiation.cc:407] Connection header received
0924 07:09:11.513745 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:11.513748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:11.513796 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:11.513908 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:11.514166 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.514861 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.516907 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.517545 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.517688 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:11.517775 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:11.518027 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 07:09:11.518108 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 07:09:11.518265 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:11.512999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:59193 (local address 127.3.217.1:34653)
0924 07:09:11.513282 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:11.513286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:11.513303 (+ 17us) server_negotiation.cc:407] Connection header received
0924 07:09:11.513476 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:11.513480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:11.513550 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:11.513691 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:11.514015 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.514808 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.516907 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.517690 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.517793 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:11.517963 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:11.518009 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 07:09:11.518102 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":107,"threads_started":1}
I20260924 07:09:11.518296 20320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:11.512948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:34653 (local address 127.3.217.3:59193)
0924 07:09:11.513218 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:11.513237 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:11.513350 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:11.513712 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:11.513716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:11.513753 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:11.513884 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:11.513891 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.515018 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.515023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:11.516698 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:11.516713 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.517822 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.517826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:11.517923 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:11.517940 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 07:09:11.517994 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 07:09:11.518098 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":198}
W20260924 07:09:11.518656 20317 tablet.cc:2369] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:09:11.518302 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:11.512972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:38221 (local address 127.3.217.3:52737)
0924 07:09:11.513475 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:11.513490 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:11.513613 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:11.513933 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:11.513935 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:11.513953 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:11.514062 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:11.514068 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.515018 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.515023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:11.516698 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:11.516713 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.517702 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.517705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:11.517817 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:11.517892 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 07:09:11.517988 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 07:09:11.518098 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":215,"thread_start_us":127,"threads_started":1}
I20260924 07:09:11.519233 20000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf1afd2b5ad1476489d3db5e5271343e" candidate_uuid: "a2e4050af6ed46daafc8bb828b32d218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" is_pre_election: true
I20260924 07:09:11.519562 20000 raft_consensus.cc:2454] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e4050af6ed46daafc8bb828b32d218 in term 0.
I20260924 07:09:11.519630 20138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf1afd2b5ad1476489d3db5e5271343e" candidate_uuid: "a2e4050af6ed46daafc8bb828b32d218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6088e628e91d476985b2dce6dbc76d2f" is_pre_election: true
I20260924 07:09:11.519918 20138 raft_consensus.cc:2454] T cf1afd2b5ad1476489d3db5e5271343e P 6088e628e91d476985b2dce6dbc76d2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e4050af6ed46daafc8bb828b32d218 in term 0.
I20260924 07:09:11.520157 20203 leader_election.cc:302] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2e4050af6ed46daafc8bb828b32d218, dd76bdd5c0bf4b93a9aa8a8022b3ebb0; no voters:
I20260924 07:09:11.520433 20333 raft_consensus.cc:2781] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:09:11.520509 20333 raft_consensus.cc:482] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:09:11.520539 20333 raft_consensus.cc:3037] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:11.521543 20333 raft_consensus.cc:504] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:09:11.521966 20333 leader_election.cc:288] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218 [CANDIDATE]: Term 1 election: Requested vote from peers 6088e628e91d476985b2dce6dbc76d2f (127.3.217.2:38221), dd76bdd5c0bf4b93a9aa8a8022b3ebb0 (127.3.217.1:34653)
I20260924 07:09:11.522428 20138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf1afd2b5ad1476489d3db5e5271343e" candidate_uuid: "a2e4050af6ed46daafc8bb828b32d218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6088e628e91d476985b2dce6dbc76d2f"
I20260924 07:09:11.522557 20138 raft_consensus.cc:3037] T cf1afd2b5ad1476489d3db5e5271343e P 6088e628e91d476985b2dce6dbc76d2f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:11.522604 20000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf1afd2b5ad1476489d3db5e5271343e" candidate_uuid: "a2e4050af6ed46daafc8bb828b32d218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0"
I20260924 07:09:11.522820 20000 raft_consensus.cc:3037] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:11.523622 20138 raft_consensus.cc:2454] T cf1afd2b5ad1476489d3db5e5271343e P 6088e628e91d476985b2dce6dbc76d2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e4050af6ed46daafc8bb828b32d218 in term 1.
I20260924 07:09:11.523931 20206 leader_election.cc:302] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6088e628e91d476985b2dce6dbc76d2f, a2e4050af6ed46daafc8bb828b32d218; no voters:
I20260924 07:09:11.524083 20333 raft_consensus.cc:2781] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:11.524312 20333 raft_consensus.cc:686] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218 [term 1 LEADER]: Becoming Leader. State: Replica: a2e4050af6ed46daafc8bb828b32d218, State: Running, Role: LEADER
I20260924 07:09:11.524768 20000 raft_consensus.cc:2454] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e4050af6ed46daafc8bb828b32d218 in term 1.
I20260924 07:09:11.524720 20333 consensus_queue.cc:237] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:09:11.528088 19858 catalog_manager.cc:5495] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e4050af6ed46daafc8bb828b32d218 (127.3.217.3). New cstate: current_term: 1 leader_uuid: "a2e4050af6ed46daafc8bb828b32d218" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } health_report { overall_health: UNKNOWN } } }
W20260924 07:09:11.556545 20185 tablet.cc:2369] T cf1afd2b5ad1476489d3db5e5271343e P 6088e628e91d476985b2dce6dbc76d2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:09:11.568830 20326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:11.564665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33424 (local address 127.3.217.3:41297)
0924 07:09:11.564842 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:11.564846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:11.564863 (+ 17us) server_negotiation.cc:407] Connection header received
0924 07:09:11.565130 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:11.565137 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:11.565197 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:11.565286 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:09:11.565536 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.566107 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.566930 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:11.567144 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:11.567226 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:11.567633 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:09:11.568212 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:09:11.568344 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:11.568602 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 07:09:11.568658 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 07:09:11.575170 20000 raft_consensus.cc:1261] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Refusing update from remote peer a2e4050af6ed46daafc8bb828b32d218: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:11.575649 20138 raft_consensus.cc:1261] T cf1afd2b5ad1476489d3db5e5271343e P 6088e628e91d476985b2dce6dbc76d2f [term 1 FOLLOWER]: Refusing update from remote peer a2e4050af6ed46daafc8bb828b32d218: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:11.575868 20337 consensus_queue.cc:1035] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:11.576496 20337 consensus_queue.cc:1035] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:11.582508 20340 mvcc.cc:205] Tried to move back new op lower bound from 7332797446444310528 to 7332797446248517632. Current Snapshot: MvccSnapshot[applied={T|T < 7332797446444310528}]
I20260924 07:09:11.586405 20343 mvcc.cc:205] Tried to move back new op lower bound from 7332797446444310528 to 7332797446248517632. Current Snapshot: MvccSnapshot[applied={T|T < 7332797446444310528 or (T in {7332797446444310528})}]
I20260924 07:09:11.590073 19858 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:11.590210 19858 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:11.592309 19858 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93: Sending DeleteTablet for 3 replicas of tablet cf1afd2b5ad1476489d3db5e5271343e
I20260924 07:09:11.592905 20250 tablet_service.cc:1514] Processing DeleteTablet for tablet cf1afd2b5ad1476489d3db5e5271343e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:11 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:33420
I20260924 07:09:11.592983 20118 tablet_service.cc:1514] Processing DeleteTablet for tablet cf1afd2b5ad1476489d3db5e5271343e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:11 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:52928
I20260924 07:09:11.593055 19980 tablet_service.cc:1514] Processing DeleteTablet for tablet cf1afd2b5ad1476489d3db5e5271343e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:11 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:51798
I20260924 07:09:11.593372 20351 tablet_replica.cc:330] T cf1afd2b5ad1476489d3db5e5271343e P 6088e628e91d476985b2dce6dbc76d2f: stopping tablet replica
I20260924 07:09:11.593514 20349 tablet_replica.cc:330] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218: stopping tablet replica
I20260924 07:09:11.593545 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 19896
I20260924 07:09:11.593660 20351 raft_consensus.cc:2229] T cf1afd2b5ad1476489d3db5e5271343e P 6088e628e91d476985b2dce6dbc76d2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:11.593892 20349 raft_consensus.cc:2229] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:09:11.593931 20351 raft_consensus.cc:2258] T cf1afd2b5ad1476489d3db5e5271343e P 6088e628e91d476985b2dce6dbc76d2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:11.594337 20349 raft_consensus.cc:2258] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:11.595317 20351 ts_tablet_manager.cc:1905] T cf1afd2b5ad1476489d3db5e5271343e P 6088e628e91d476985b2dce6dbc76d2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:11.595762 20349 ts_tablet_manager.cc:1905] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:11.598565 20349 ts_tablet_manager.cc:1918] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:09:11.598723 20349 log.cc:1196] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/cf1afd2b5ad1476489d3db5e5271343e
I20260924 07:09:11.598817 20351 ts_tablet_manager.cc:1918] T cf1afd2b5ad1476489d3db5e5271343e P 6088e628e91d476985b2dce6dbc76d2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:09:11.599139 20351 log.cc:1196] T cf1afd2b5ad1476489d3db5e5271343e P 6088e628e91d476985b2dce6dbc76d2f: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/cf1afd2b5ad1476489d3db5e5271343e
I20260924 07:09:11.599375 20349 ts_tablet_manager.cc:1939] T cf1afd2b5ad1476489d3db5e5271343e P a2e4050af6ed46daafc8bb828b32d218: Deleting consensus metadata
I20260924 07:09:11.599654 20351 ts_tablet_manager.cc:1939] T cf1afd2b5ad1476489d3db5e5271343e P 6088e628e91d476985b2dce6dbc76d2f: Deleting consensus metadata
I20260924 07:09:11.601027 19845 catalog_manager.cc:4841] TS 6088e628e91d476985b2dce6dbc76d2f (127.3.217.2:38221): tablet cf1afd2b5ad1476489d3db5e5271343e (table test-table [id=78a3edee33004a0eb0db1e5b246f9e75]) successfully deleted
I20260924 07:09:11.601217 19845 catalog_manager.cc:4841] TS a2e4050af6ed46daafc8bb828b32d218 (127.3.217.3:41297): tablet cf1afd2b5ad1476489d3db5e5271343e (table test-table [id=78a3edee33004a0eb0db1e5b246f9e75]) successfully deleted
W20260924 07:09:11.604831 19842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.217.1:34653 (error 108)
I20260924 07:09:11.605818 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 20050
W20260924 07:09:11.605839 20325 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:09:11.605530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:34653 (local address 127.0.0.1:51810)
0924 07:09:11.605698 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:11.605760 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.1:34653: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":89}
W20260924 07:09:11.605974 19842 catalog_manager.cc:4584] TS dd76bdd5c0bf4b93a9aa8a8022b3ebb0 (127.3.217.1:34653): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cf1afd2b5ad1476489d3db5e5271343e: Network error: Client connection negotiation failed: client connection to 127.3.217.1:34653: connect: Connection reset by peer (error 104)
I20260924 07:09:11.611560 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 20187
I20260924 07:09:11.616384 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 19825
I20260924 07:09:31.622059 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:37863
--webserver_interface=127.3.217.62
--webserver_port=36079
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:37863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:31.751160 20353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:31.751546 20353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:31.751609 20353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:31.755949 20353 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 07:09:31.756033 20353 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 07:09:31.756049 20353 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 07:09:31.756062 20353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:09:31.756081 20353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:31.756099 20353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:31.756114 20353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:09:31.756129 20353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:09:31.762109 20353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:37863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:37863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=36079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:31.763856 20353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:31.765262 20353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:31.775787 20358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:31.775830 20359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:31.775971 20353 server_base.cc:1034] running on GCE node
W20260924 07:09:31.775830 20361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:31.776839 20353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:31.778228 20353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:31.779436 20353 hybrid_clock.cc:648] HybridClock initialized: now 1790233771779395 us; error 47 us; skew 500 ppm
Sep 24 07:09:31 dist-test-slave-2p7j krb5kdc[19277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233771, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:31.787286 20353 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:09:31.788601 20353 webserver.cc:460] Webserver started at http://127.3.217.62:36079/ using document root <none> and password file <none>
I20260924 07:09:31.789300 20353 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:31.789381 20353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:31.794147 20353 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260924 07:09:31.796872 20368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:31.798543 20353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:09:31.798697 20353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "011e0254f5814c418c1210dd0c6a4c93"
format_stamp: "Formatted at 2026-09-24 07:08:49 on dist-test-slave-2p7j"
server_key: "f328d2cae7e9a9a79c21825510c92360"
server_key_iv: "3f7b447fc44f7d9a4b973eeb1da0b67f"
server_key_version: "encryptionkey@0"
I20260924 07:09:31.799386 20353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:31.812387 20353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:31.816079 20353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:31.816296 20353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:31.825018 20353 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:37863
I20260924 07:09:31.825306 20420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:37863 every 8 connection(s)
I20260924 07:09:31.826431 20353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:09:31.829589 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 20353
I20260924 07:09:31.830065 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:34653
--local_ip_for_outbound_sockets=127.3.217.1
--tserver_master_addrs=127.3.217.62:37863
--webserver_port=45069
--webserver_interface=127.3.217.1
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:09:31.831493 20421 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:09:31.833688 20421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93: Bootstrap starting.
I20260924 07:09:31.851032 20421 log.cc:824] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:31.862610 20421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:09:31.862946 20421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93: Bootstrap complete.
I20260924 07:09:31.866856 20421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011e0254f5814c418c1210dd0c6a4c93" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37863 } }
I20260924 07:09:31.867246 20421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 011e0254f5814c418c1210dd0c6a4c93, State: Initialized, Role: FOLLOWER
I20260924 07:09:31.867776 20421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011e0254f5814c418c1210dd0c6a4c93" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37863 } }
I20260924 07:09:31.867901 20421 raft_consensus.cc:388] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:09:31.867970 20421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:09:31.868063 20421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 2 FOLLOWER]: Advancing to term 3
I20260924 07:09:31.869925 20421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011e0254f5814c418c1210dd0c6a4c93" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37863 } }
I20260924 07:09:31.870235 20421 leader_election.cc:302] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 011e0254f5814c418c1210dd0c6a4c93; no voters:
I20260924 07:09:31.870523 20421 leader_election.cc:288] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 07:09:31.870615 20425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 07:09:31.870855 20425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [term 3 LEADER]: Becoming Leader. State: Replica: 011e0254f5814c418c1210dd0c6a4c93, State: Running, Role: LEADER
I20260924 07:09:31.871253 20425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011e0254f5814c418c1210dd0c6a4c93" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37863 } }
I20260924 07:09:31.871796 20421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:09:31.872812 20426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "011e0254f5814c418c1210dd0c6a4c93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011e0254f5814c418c1210dd0c6a4c93" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37863 } } }
I20260924 07:09:31.873032 20426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:31.873528 20427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 011e0254f5814c418c1210dd0c6a4c93. Latest consensus state: current_term: 3 leader_uuid: "011e0254f5814c418c1210dd0c6a4c93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011e0254f5814c418c1210dd0c6a4c93" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37863 } } }
I20260924 07:09:31.873697 20427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:31.874050 20434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:09:31.879180 20434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:09:31.879853 20434 catalog_manager.cc:1259] Loaded cluster ID: de6892e2dc1940018ef0a59b3f8ae3a4
I20260924 07:09:31.879917 20434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:09:31.882829 20434 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:09:31.883623 20434 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93: Loaded TSK: 0
I20260924 07:09:31.883711 20434 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 011e0254f5814c418c1210dd0c6a4c93: Loaded TSK: 1
F20260924 07:09:31.885052 20434 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f381ab519ad google::LogMessage::Fail() at ??:0
@ 0x7f381ab5488a google::LogMessage::SendToLog() at ??:0
@ 0x7f381ab514de google::LogMessage::Flush() at ??:0
@ 0x7f381ab52a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f381d81518d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f381d80df0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f38254b67bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f382549099c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f3825491178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f38254d2df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f38254f182b std::function<>::operator()() at ??:0
@ 0x7f3825490b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f3825491f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f382548e579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f38254d2640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f381cda33d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f381cda3c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f381cda53f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f381cd94512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f381e3d86db start_thread at ??:0
@ 0x7f381964f61f clone at ??:0
W20260924 07:09:31.998270 20423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:31.998608 20423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:31.998659 20423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:32.003487 20423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:32.003664 20423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:32.003758 20423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:09:32.010562 20423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:34653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=45069
--tserver_master_addrs=127.3.217.62:37863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:32.012182 20423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:32.013532 20423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:32.025415 20468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:32.025410 20469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:32.025422 20471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:32.025938 20423 server_base.cc:1034] running on GCE node
I20260924 07:09:32.026414 20423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:32.027230 20423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:32.028410 20423 hybrid_clock.cc:648] HybridClock initialized: now 1790233772028366 us; error 81 us; skew 500 ppm
Sep 24 07:09:32 dist-test-slave-2p7j krb5kdc[19277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:32.031845 20423 init.cc:377] Logged in from keytab as kudu/127.3.217.1@KRBTEST.COM (short username kudu)
I20260924 07:09:32.033188 20423 webserver.cc:460] Webserver started at http://127.3.217.1:45069/ using document root <none> and password file <none>
I20260924 07:09:32.033972 20423 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:32.034065 20423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:32.040989 20423 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 07:09:32.044054 20478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:32.045888 20423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:09:32.046031 20423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0"
format_stamp: "Formatted at 2026-09-24 07:08:49 on dist-test-slave-2p7j"
server_key: "0e812f3b4e4d176daee294c07b1764c8"
server_key_iv: "cb4db9aa3133272b918111d4418da5db"
server_key_version: "encryptionkey@0"
I20260924 07:09:32.046816 20423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:32.065338 20423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:32.069293 20423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:32.069504 20423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:32.070230 20423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:32.071691 20485 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 07:09:32.076761 20423 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 07:09:32.076848 20423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260924 07:09:32.076918 20423 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 07:09:32.079358 20423 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 07:09:32.079421 20423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260924 07:09:32.079948 20485 tablet_bootstrap.cc:492] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Bootstrap starting.
I20260924 07:09:32.090852 20423 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:34653
I20260924 07:09:32.091076 20592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:34653 every 8 connection(s)
I20260924 07:09:32.092159 20423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:09:32.099365 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 20423
I20260924 07:09:32.100001 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:38221
--local_ip_for_outbound_sockets=127.3.217.2
--tserver_master_addrs=127.3.217.62:37863
--webserver_port=34709
--webserver_interface=127.3.217.2
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:09:32.104105 20485 log.cc:824] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Log is configured to *not* fsync() on all Append() calls
W20260924 07:09:32.105181 20595 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:09:32.094986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.1:50739)
0924 07:09:32.095741 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:32.095829 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:32.097302 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:32.104093 (+ 6791us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":173,"threads_started":1}
W20260924 07:09:32.105690 20593 heartbeater.cc:644] Failed to heartbeat to 127.3.217.62:37863 (0 consecutive failures): Network error: Failed to ping master at 127.3.217.62:37863: Client connection negotiation failed: client connection to 127.3.217.62:37863: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 07:09:32.106595 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:32.106309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.1:46823)
0924 07:09:32.106451 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:32.106523 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 07:09:32.107337 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:32.107136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.1:36971)
0924 07:09:32.107238 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:32.107283 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 07:09:32.107575 20593 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 07:09:32.113823 20485 tablet_bootstrap.cc:492] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:09:32.114234 20485 tablet_bootstrap.cc:492] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Bootstrap complete.
I20260924 07:09:32.115619 20485 ts_tablet_manager.cc:1397] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Time spent bootstrapping tablet: real 0.036s user 0.020s sys 0.015s
I20260924 07:09:32.118709 20485 raft_consensus.cc:348] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:09:32.119251 20485 raft_consensus.cc:729] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd76bdd5c0bf4b93a9aa8a8022b3ebb0, State: Initialized, Role: FOLLOWER
I20260924 07:09:32.119783 20485 consensus_queue.cc:260] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:09:32.120849 20485 ts_tablet_manager.cc:1428] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:09:32.121403 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:32.121105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.1:39263)
0924 07:09:32.121239 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:32.121305 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 07:09:32.122313 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:32.122032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.1:60621)
0924 07:09:32.122170 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:32.122226 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 07:09:32.223055 20596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:32.223284 20596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:32.223338 20596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:32.227304 20596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:32.227360 20596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:32.227434 20596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:09:32.232430 20596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:38221
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=34709
--tserver_master_addrs=127.3.217.62:37863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:32.233389 20596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:32.234333 20596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:32.244069 20606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:32.244067 20603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:32.244114 20604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:32.244457 20596 server_base.cc:1034] running on GCE node
I20260924 07:09:32.245045 20596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:32.245784 20596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:32.246996 20596 hybrid_clock.cc:648] HybridClock initialized: now 1790233772246972 us; error 37 us; skew 500 ppm
Sep 24 07:09:32 dist-test-slave-2p7j krb5kdc[19277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:32.250272 20596 init.cc:377] Logged in from keytab as kudu/127.3.217.2@KRBTEST.COM (short username kudu)
I20260924 07:09:32.251362 20596 webserver.cc:460] Webserver started at http://127.3.217.2:34709/ using document root <none> and password file <none>
I20260924 07:09:32.252146 20596 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:32.252221 20596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:32.256464 20596 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:09:32.258790 20613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:32.260241 20596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:09:32.260345 20596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "6088e628e91d476985b2dce6dbc76d2f"
format_stamp: "Formatted at 2026-09-24 07:08:49 on dist-test-slave-2p7j"
server_key: "73073cb316d13edec636e332d72a3462"
server_key_iv: "ecfd326d53ddba8e0d6e598b9c3d2b47"
server_key_version: "encryptionkey@0"
I20260924 07:09:32.260880 20596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:32.281839 20596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:32.285117 20596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:32.285302 20596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:32.286007 20596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:32.286863 20596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:32.286914 20596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:32.286966 20596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:32.286995 20596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:32.297220 20596 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:38221
I20260924 07:09:32.297292 20726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:38221 every 8 connection(s)
I20260924 07:09:32.298247 20596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:09:32.299448 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 20596
I20260924 07:09:32.299914 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:41297
--local_ip_for_outbound_sockets=127.3.217.3
--tserver_master_addrs=127.3.217.62:37863
--webserver_port=41719
--webserver_interface=127.3.217.3
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:09:32.303078 20730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:32.301086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.2:60361)
0924 07:09:32.302002 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:32.302169 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":156,"threads_started":1}
W20260924 07:09:32.303642 20727 heartbeater.cc:644] Failed to heartbeat to 127.3.217.62:37863 (0 consecutive failures): Network error: Failed to ping master at 127.3.217.62:37863: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
I20260924 07:09:32.304380 20730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:32.304131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.2:51759)
0924 07:09:32.304254 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:32.304319 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 07:09:32.304971 20730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:32.304776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.2:40521)
0924 07:09:32.304875 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:32.304919 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 07:09:32.305127 20727 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:09:32.405802 20729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:32.406042 20729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:32.406097 20729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:32.410018 20729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:32.410071 20729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:32.410125 20729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:09:32.415036 20729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:41297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=41719
--tserver_master_addrs=127.3.217.62:37863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:32.415943 20729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:32.416882 20729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:32.425830 20736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:32.425824 20735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:32.425825 20738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:32.425973 20729 server_base.cc:1034] running on GCE node
I20260924 07:09:32.426844 20729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:32.427439 20729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:32.428629 20729 hybrid_clock.cc:648] HybridClock initialized: now 1790233772428601 us; error 40 us; skew 500 ppm
Sep 24 07:09:32 dist-test-slave-2p7j krb5kdc[19277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:32.431282 20729 init.cc:377] Logged in from keytab as kudu/127.3.217.3@KRBTEST.COM (short username kudu)
I20260924 07:09:32.432104 20729 webserver.cc:460] Webserver started at http://127.3.217.3:41719/ using document root <none> and password file <none>
I20260924 07:09:32.432629 20729 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:32.432695 20729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:32.436291 20729 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:09:32.438220 20745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:32.439584 20729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:09:32.439689 20729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "a2e4050af6ed46daafc8bb828b32d218"
format_stamp: "Formatted at 2026-09-24 07:08:50 on dist-test-slave-2p7j"
server_key: "22c21432ac0b16076fde3a1d97de8393"
server_key_iv: "13355559d3de564e50607a5b788a8647"
server_key_version: "encryptionkey@0"
I20260924 07:09:32.440220 20729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:32.451800 20729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:32.454299 20729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:32.454450 20729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:32.454941 20729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:32.455682 20729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:32.455736 20729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:32.455787 20729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:32.455814 20729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:32.463990 20729 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:41297
I20260924 07:09:32.464027 20858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:41297 every 8 connection(s)
I20260924 07:09:32.464952 20729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:09:32.467128 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 20729
I20260924 07:09:32.469310 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:32.466947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.3:34935)
0924 07:09:32.468570 (+ 1623us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:32.468664 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1446,"thread_start_us":113,"threads_started":1}
W20260924 07:09:32.469679 20859 heartbeater.cc:644] Failed to heartbeat to 127.3.217.62:37863 (0 consecutive failures): Network error: Failed to ping master at 127.3.217.62:37863: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
I20260924 07:09:32.470176 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:32.469972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.3:43009)
0924 07:09:32.470073 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:32.470126 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 07:09:32.470674 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:32.470495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.3:38477)
0924 07:09:32.470584 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:32.470626 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 07:09:32.470818 20859 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 07:09:33.123555 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:33.123068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.1:49703)
0924 07:09:33.123343 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:33.123402 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":73,"threads_started":1}
I20260924 07:09:33.306042 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:33.305652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.2:33103)
0924 07:09:33.305891 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:33.305947 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":71,"threads_started":1}
I20260924 07:09:33.472043 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:33.471549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.3:42123)
0924 07:09:33.471876 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:33.471945 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":98,"threads_started":1}
I20260924 07:09:33.514053 20866 raft_consensus.cc:482] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:09:33.514350 20866 raft_consensus.cc:504] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:09:33.515770 20866 leader_election.cc:288] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2e4050af6ed46daafc8bb828b32d218 (127.3.217.3:41297), 6088e628e91d476985b2dce6dbc76d2f (127.3.217.2:38221)
Sep 24 07:09:33 dist-test-slave-2p7j krb5kdc[19277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.3@KRBTEST.COM
Sep 24 07:09:33 dist-test-slave-2p7j krb5kdc[19277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.2@KRBTEST.COM
I20260924 07:09:33.530340 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:33.516638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:47573 (local address 127.3.217.2:38221)
0924 07:09:33.517059 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:33.517086 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:33.517133 (+ 47us) server_negotiation.cc:407] Connection header received
0924 07:09:33.517929 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:33.517936 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:33.518320 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:33.518618 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:33.519675 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:33.520733 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:33.521943 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:33.522443 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:33.522611 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:33.525873 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:33.525881 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:33.525895 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:33.525941 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:33.527713 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:33.528445 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:33.528451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:33.528456 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:33.528570 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:33.528879 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:33.528882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:33.528884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:33.529276 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:33.529394 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:33.529894 (+ 500us) server_negotiation.cc:299] Negotiation successful
0924 07:09:33.530149 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":118,"threads_started":1}
I20260924 07:09:33.530426 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:33.516281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:38221 (local address 127.3.217.1:47573)
0924 07:09:33.516601 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:33.516618 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:33.516792 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:33.518579 (+ 1787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:33.518583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:33.518898 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:33.519460 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:33.519480 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:33.520869 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:33.520873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:33.521820 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:33.521830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:33.522563 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:33.522566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:33.522834 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:33.523619 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:33.523630 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:33.525568 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:33.527839 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:33.527843 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:33.527855 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:33.528337 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:33.528659 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:33.528662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:33.528664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:33.528791 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:33.529386 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:33.529392 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:33.529603 (+ 211us) client_negotiation.cc:769] Sending connection context
0924 07:09:33.529770 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 07:09:33.530139 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":146,"spinlock_wait_cycles":37632,"thread_start_us":74,"threads_started":1}
I20260924 07:09:33.530447 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:33.515998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:41297 (local address 127.3.217.1:44343)
0924 07:09:33.516127 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:33.516142 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:33.516251 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:33.518372 (+ 2121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:33.518379 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:33.518890 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:33.519459 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:33.519477 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:33.520791 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:33.520794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:33.521874 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:33.521884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:33.522592 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:33.522595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:33.522836 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:33.523488 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:33.523537 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:33.525306 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:33.527825 (+ 2519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:33.527832 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:33.527853 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:33.528341 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:33.528788 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:33.528791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:33.528792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:33.528867 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:33.529537 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:33.529540 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:33.529642 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 07:09:33.529771 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 07:09:33.530227 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":336}
I20260924 07:09:33.530447 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:33.516180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:44343 (local address 127.3.217.3:41297)
0924 07:09:33.516710 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:33.516744 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:33.516799 (+ 55us) server_negotiation.cc:407] Connection header received
0924 07:09:33.517633 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:33.517640 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:33.518007 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:33.518301 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:33.519656 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:33.520649 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:33.522005 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:33.522477 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:33.522631 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:33.525509 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:33.525514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:33.525523 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:33.525551 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:33.527677 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:33.528514 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:33.528521 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:33.528525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:33.528687 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:33.528953 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:33.528956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:33.528958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:33.529424 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:33.529561 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:33.529916 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 07:09:33.530227 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":118,"threads_started":1}
W20260924 07:09:33.533354 20482 leader_election.cc:341] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2e4050af6ed46daafc8bb828b32d218 (127.3.217.3:41297): Not found: Tablet not found: cf1afd2b5ad1476489d3db5e5271343e
W20260924 07:09:33.533583 20482 leader_election.cc:341] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6088e628e91d476985b2dce6dbc76d2f (127.3.217.2:38221): Not found: Tablet not found: cf1afd2b5ad1476489d3db5e5271343e
I20260924 07:09:33.533643 20482 leader_election.cc:302] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd76bdd5c0bf4b93a9aa8a8022b3ebb0; no voters: 6088e628e91d476985b2dce6dbc76d2f, a2e4050af6ed46daafc8bb828b32d218
I20260924 07:09:33.533941 20866 raft_consensus.cc:2726] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:09:34.125171 20870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:34.124650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.1:51107)
0924 07:09:34.124961 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:34.125044 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":80,"threads_started":1}
I20260924 07:09:34.307447 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:34.306916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.2:39697)
0924 07:09:34.307278 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:34.307340 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":142,"threads_started":1}
I20260924 07:09:34.473322 20872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:34.472868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.3:45057)
0924 07:09:34.473166 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:34.473226 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":96,"threads_started":1}
I20260924 07:09:35.126652 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:35.126016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.1:43805)
0924 07:09:35.126430 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:35.126520 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":85,"threads_started":1}
I20260924 07:09:35.225898 20874 raft_consensus.cc:482] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:09:35.226114 20874 raft_consensus.cc:504] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:09:35.226660 20874 leader_election.cc:288] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2e4050af6ed46daafc8bb828b32d218 (127.3.217.3:41297), 6088e628e91d476985b2dce6dbc76d2f (127.3.217.2:38221)
W20260924 07:09:35.227692 20482 leader_election.cc:341] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2e4050af6ed46daafc8bb828b32d218 (127.3.217.3:41297): Not found: Tablet not found: cf1afd2b5ad1476489d3db5e5271343e
W20260924 07:09:35.227890 20482 leader_election.cc:341] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6088e628e91d476985b2dce6dbc76d2f (127.3.217.2:38221): Not found: Tablet not found: cf1afd2b5ad1476489d3db5e5271343e
I20260924 07:09:35.227952 20482 leader_election.cc:302] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd76bdd5c0bf4b93a9aa8a8022b3ebb0; no voters: 6088e628e91d476985b2dce6dbc76d2f, a2e4050af6ed46daafc8bb828b32d218
I20260924 07:09:35.228102 20874 raft_consensus.cc:2726] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:09:35.309020 20875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:35.308426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.2:42921)
0924 07:09:35.308825 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:35.308895 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":97,"threads_started":1}
I20260924 07:09:35.474716 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:35.474182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.3:42831)
0924 07:09:35.474513 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:35.474583 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":109,"threads_started":1}
I20260924 07:09:36.128217 20877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:36.127677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.1:36733)
0924 07:09:36.128021 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:36.128087 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":96,"threads_started":1}
I20260924 07:09:36.310525 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:36.309985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.2:55441)
0924 07:09:36.310340 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:36.310414 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":109,"threads_started":1}
I20260924 07:09:36.476107 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:36.475632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.3:54675)
0924 07:09:36.475934 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:36.475997 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":95,"threads_started":1}
I20260924 07:09:36.737743 20880 raft_consensus.cc:482] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:09:36.737890 20880 raft_consensus.cc:504] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:09:36.738291 20880 leader_election.cc:288] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2e4050af6ed46daafc8bb828b32d218 (127.3.217.3:41297), 6088e628e91d476985b2dce6dbc76d2f (127.3.217.2:38221)
W20260924 07:09:36.739256 20482 leader_election.cc:341] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2e4050af6ed46daafc8bb828b32d218 (127.3.217.3:41297): Not found: Tablet not found: cf1afd2b5ad1476489d3db5e5271343e
W20260924 07:09:36.739473 20482 leader_election.cc:341] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6088e628e91d476985b2dce6dbc76d2f (127.3.217.2:38221): Not found: Tablet not found: cf1afd2b5ad1476489d3db5e5271343e
I20260924 07:09:36.739567 20482 leader_election.cc:302] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd76bdd5c0bf4b93a9aa8a8022b3ebb0; no voters: 6088e628e91d476985b2dce6dbc76d2f, a2e4050af6ed46daafc8bb828b32d218
I20260924 07:09:36.739727 20880 raft_consensus.cc:2726] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:09:37.129844 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:37.129308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.1:36961)
0924 07:09:37.129649 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:37.129721 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":78,"threads_started":1}
I20260924 07:09:37.312252 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:37.311666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.2:56231)
0924 07:09:37.312040 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:37.312115 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":124,"threads_started":1}
I20260924 07:09:37.477516 20883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:37.477015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.3:57027)
0924 07:09:37.477349 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:37.477410 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":128,"threads_started":1}
I20260924 07:09:38.131409 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:38.130821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.1:44873)
0924 07:09:38.131187 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:38.131277 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":100,"threads_started":1}
I20260924 07:09:38.313848 20885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:38.313280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.2:58077)
0924 07:09:38.313628 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:38.313716 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":87,"threads_started":1}
I20260924 07:09:38.478781 20886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:38.478269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.3:37007)
0924 07:09:38.478575 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:38.478650 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":74,"threads_started":1}
I20260924 07:09:38.814743 20887 raft_consensus.cc:482] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:09:38.814894 20887 raft_consensus.cc:504] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:09:38.815311 20887 leader_election.cc:288] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2e4050af6ed46daafc8bb828b32d218 (127.3.217.3:41297), 6088e628e91d476985b2dce6dbc76d2f (127.3.217.2:38221)
W20260924 07:09:38.816241 20482 leader_election.cc:341] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2e4050af6ed46daafc8bb828b32d218 (127.3.217.3:41297): Not found: Tablet not found: cf1afd2b5ad1476489d3db5e5271343e
W20260924 07:09:38.816370 20482 leader_election.cc:341] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6088e628e91d476985b2dce6dbc76d2f (127.3.217.2:38221): Not found: Tablet not found: cf1afd2b5ad1476489d3db5e5271343e
I20260924 07:09:38.816428 20482 leader_election.cc:302] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd76bdd5c0bf4b93a9aa8a8022b3ebb0; no voters: 6088e628e91d476985b2dce6dbc76d2f, a2e4050af6ed46daafc8bb828b32d218
I20260924 07:09:38.816670 20887 raft_consensus.cc:2726] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:09:39.133191 20888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:39.132466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.1:59113)
0924 07:09:39.132836 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:39.132913 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":102,"threads_started":1}
I20260924 07:09:39.315461 20889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:39.314805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.2:52579)
0924 07:09:39.315204 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:39.315284 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":88,"threads_started":1}
I20260924 07:09:39.480345 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:39.479785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.3:56329)
0924 07:09:39.480163 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:39.480237 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":101,"threads_started":1}
I20260924 07:09:40.134626 20891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:40.133986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.1:33027)
0924 07:09:40.134377 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:40.134472 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":74,"threads_started":1}
I20260924 07:09:40.316675 20892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:40.316252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.2:41427)
0924 07:09:40.316516 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:40.316573 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
I20260924 07:09:40.481401 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:40.481030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.3:43097)
0924 07:09:40.481258 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:40.481314 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":72,"threads_started":1}
I20260924 07:09:41.136169 20894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:41.135637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.1:43799)
0924 07:09:41.135958 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:41.136044 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":116,"threads_started":1}
I20260924 07:09:41.318169 20895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:41.317622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.2:38953)
0924 07:09:41.317985 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:41.318061 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":129,"threads_started":1}
I20260924 07:09:41.482918 20896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:41.482312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.3:40279)
0924 07:09:41.482706 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:41.482792 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":122,"threads_started":1}
I20260924 07:09:41.967164 20897 raft_consensus.cc:482] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:09:41.967406 20897 raft_consensus.cc:504] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e4050af6ed46daafc8bb828b32d218" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 41297 } } peers { permanent_uuid: "6088e628e91d476985b2dce6dbc76d2f" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 38221 } } peers { permanent_uuid: "dd76bdd5c0bf4b93a9aa8a8022b3ebb0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 34653 } }
I20260924 07:09:41.968003 20897 leader_election.cc:288] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2e4050af6ed46daafc8bb828b32d218 (127.3.217.3:41297), 6088e628e91d476985b2dce6dbc76d2f (127.3.217.2:38221)
W20260924 07:09:41.968770 20482 leader_election.cc:341] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6088e628e91d476985b2dce6dbc76d2f (127.3.217.2:38221): Not found: Tablet not found: cf1afd2b5ad1476489d3db5e5271343e
W20260924 07:09:41.968928 20482 leader_election.cc:341] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2e4050af6ed46daafc8bb828b32d218 (127.3.217.3:41297): Not found: Tablet not found: cf1afd2b5ad1476489d3db5e5271343e
I20260924 07:09:41.968977 20482 leader_election.cc:302] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd76bdd5c0bf4b93a9aa8a8022b3ebb0; no voters: 6088e628e91d476985b2dce6dbc76d2f, a2e4050af6ed46daafc8bb828b32d218
I20260924 07:09:41.969123 20897 raft_consensus.cc:2726] T cf1afd2b5ad1476489d3db5e5271343e P dd76bdd5c0bf4b93a9aa8a8022b3ebb0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:09:42.137624 20898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.137119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.1:52655)
0924 07:09:42.137413 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:42.137491 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":90,"threads_started":1}
I20260924 07:09:42.319492 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.318908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.2:47215)
0924 07:09:42.319274 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:42.319364 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":77,"threads_started":1}
I20260924 07:09:42.484820 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.484135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.3:58915)
0924 07:09:42.484494 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:42.484581 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":99,"threads_started":1}
I20260924 07:09:43.139148 20901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:43.138472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.1:37285)
0924 07:09:43.138912 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:43.139015 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":135,"threads_started":1}
I20260924 07:09:43.320955 20902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:43.320398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.2:57143)
0924 07:09:43.320783 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:43.320852 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":98,"threads_started":1}
I20260924 07:09:43.486265 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:43.485754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.3:41343)
0924 07:09:43.486065 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:43.486137 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":90,"threads_started":1}
I20260924 07:09:44.140796 20904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.140216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.1:38469)
0924 07:09:44.140605 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:44.140673 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":114,"threads_started":1}
I20260924 07:09:44.322549 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.321998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.2:45275)
0924 07:09:44.322364 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:44.322440 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":101,"threads_started":1}
I20260924 07:09:44.487664 20906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.486983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.3:52295)
0924 07:09:44.487358 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:44.487529 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":90,"threads_started":1}
I20260924 07:09:45.142072 20907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:45.141655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.1:50505)
0924 07:09:45.141928 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:45.141980 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":77,"threads_started":1}
I20260924 07:09:45.324131 20908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:45.323419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.2:38871)
0924 07:09:45.323829 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:45.323916 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":106,"threads_started":1}
I20260924 07:09:45.488894 20909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:45.488504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.3:37793)
0924 07:09:45.488743 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:45.488799 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260924 07:09:46.143810 20910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:46.143204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.1:60131)
0924 07:09:46.143585 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:46.143675 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":90,"threads_started":1}
I20260924 07:09:46.325767 20911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:46.325071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.2:58499)
0924 07:09:46.325446 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:46.325624 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":126,"threads_started":1}
I20260924 07:09:46.490381 20912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:46.489799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.3:54531)
0924 07:09:46.490158 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:46.490245 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":110,"threads_started":1}
I20260924 07:09:47.145498 20913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:47.145018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.1:54671)
0924 07:09:47.145336 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:47.145398 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":95,"threads_started":1}
I20260924 07:09:47.327348 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:47.326869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37863 (local address 127.3.217.2:46069)
0924 07:09:47.327156 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:47.327225 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.62:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":82,"threads_started":1}
I20260924 07:09:47.468147 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 20423
I20260924 07:09:47.474869 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 20596
I20260924 07:09:47.480809 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 20729
2026-09-24T07:09:47Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59337 ms)
[----------] 30 tests from SecurityITest (246420 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:09:47 dist-test-slave-2p7j krb5kdc[20917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:09:47 dist-test-slave-2p7j krb5kdc[20917](info): set up 2 sockets
Sep 24 07:09:47 dist-test-slave-2p7j krb5kdc[20917](info): commencing operation
krb5kdc: starting...
W20260924 07:09:48.451665 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:09:48 dist-test-slave-2p7j krb5kdc[20917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:09:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:09:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:09:48.572407 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:34005
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:46213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:34005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:48.676293 20930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:48.676527 20930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:48.676578 20930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:48.680456 20930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:09:48.680502 20930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:48.680527 20930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:48.680547 20930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:09:48.680574 20930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:09:48.685550 20930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:46213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:34005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:34005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:48.686442 20930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:48.687359 20930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:09:48.695667 20930 server_base.cc:1034] running on GCE node
W20260924 07:09:48.695622 20935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:48.695626 20936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:48.695639 20938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:48.696382 20930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:48.697402 20930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:48.698623 20930 hybrid_clock.cc:648] HybridClock initialized: now 1790233788698555 us; error 36 us; skew 500 ppm
Sep 24 07:09:48 dist-test-slave-2p7j krb5kdc[20917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233788, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:48.701277 20930 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:09:48.702056 20930 webserver.cc:460] Webserver started at http://127.3.217.62:46095/ using document root <none> and password file <none>
I20260924 07:09:48.702540 20930 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:48.702601 20930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:48.702782 20930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:48.704689 20930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "02a4ed4653f24b79a0a70cf31e4902a2"
format_stamp: "Formatted at 2026-09-24 07:09:48 on dist-test-slave-2p7j"
server_key: "c8ff31f15c7d98133887fea603c16777"
server_key_iv: "a329f729ffadfd4c4dc7f389283f95f3"
server_key_version: "encryptionkey@0"
I20260924 07:09:48.705189 20930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "02a4ed4653f24b79a0a70cf31e4902a2"
format_stamp: "Formatted at 2026-09-24 07:09:48 on dist-test-slave-2p7j"
server_key: "c8ff31f15c7d98133887fea603c16777"
server_key_iv: "a329f729ffadfd4c4dc7f389283f95f3"
server_key_version: "encryptionkey@0"
I20260924 07:09:48.708528 20930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 07:09:48.711187 20945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:48.712466 20930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:09:48.712584 20930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "02a4ed4653f24b79a0a70cf31e4902a2"
format_stamp: "Formatted at 2026-09-24 07:09:48 on dist-test-slave-2p7j"
server_key: "c8ff31f15c7d98133887fea603c16777"
server_key_iv: "a329f729ffadfd4c4dc7f389283f95f3"
server_key_version: "encryptionkey@0"
I20260924 07:09:48.712690 20930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:48.725544 20930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:48.728542 20930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:48.728742 20930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:48.735234 20930 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:34005
I20260924 07:09:48.735273 20997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:34005 every 8 connection(s)
I20260924 07:09:48.736123 20930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:09:48.737802 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 20930
I20260924 07:09:48.737927 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:09:48.738240 3940 external_mini_cluster.cc:1428] Setting key e2d51bdb7657b23912add48c29eb4d5d
I20260924 07:09:48.739084 20998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:48.746104 20998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2: Bootstrap starting.
Sep 24 07:09:48 dist-test-slave-2p7j krb5kdc[20917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:09:48.748381 20998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:48.749238 20998 log.cc:824] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:48.751252 20998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2: No bootstrap required, opened a new log
I20260924 07:09:48.753083 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:48.739566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41144 (local address 127.3.217.62:34005)
0924 07:09:48.740131 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:48.740149 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:48.740191 (+ 42us) server_negotiation.cc:407] Connection header received
0924 07:09:48.741338 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:48.741359 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:48.741740 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:48.742083 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:48.743001 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:48.744490 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:48.745379 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:48.746038 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:48.746202 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:48.748429 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:48.748436 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:48.748451 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:48.748481 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:48.750281 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:48.750670 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:48.750675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:48.750680 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:48.750808 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:48.751062 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:48.751065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:48.751066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:48.751558 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:48.751744 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:48.751949 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 07:09:48.752225 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":225,"threads_started":1}
I20260924 07:09:48.755270 20998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a4ed4653f24b79a0a70cf31e4902a2" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 34005 } }
I20260924 07:09:48.755462 20998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:48.755496 20998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02a4ed4653f24b79a0a70cf31e4902a2, State: Initialized, Role: FOLLOWER
I20260924 07:09:48.755924 20998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a4ed4653f24b79a0a70cf31e4902a2" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 34005 } }
I20260924 07:09:48.756047 20998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:09:48.756102 20998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:09:48.756192 20998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:48.756963 20998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a4ed4653f24b79a0a70cf31e4902a2" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 34005 } }
I20260924 07:09:48.757263 20998 leader_election.cc:302] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02a4ed4653f24b79a0a70cf31e4902a2; no voters:
I20260924 07:09:48.757513 20998 leader_election.cc:288] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:09:48.757639 21003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:48.757838 21003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2 [term 1 LEADER]: Becoming Leader. State: Replica: 02a4ed4653f24b79a0a70cf31e4902a2, State: Running, Role: LEADER
I20260924 07:09:48.758214 21003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a4ed4653f24b79a0a70cf31e4902a2" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 34005 } }
I20260924 07:09:48.758525 20998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:09:48.760011 21004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02a4ed4653f24b79a0a70cf31e4902a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a4ed4653f24b79a0a70cf31e4902a2" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 34005 } } }
I20260924 07:09:48.760157 21004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:48.760398 21005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02a4ed4653f24b79a0a70cf31e4902a2. Latest consensus state: current_term: 1 leader_uuid: "02a4ed4653f24b79a0a70cf31e4902a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a4ed4653f24b79a0a70cf31e4902a2" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 34005 } } }
I20260924 07:09:48.760488 21005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2 [sys.catalog]: This master's current role is: LEADER
W20260924 07:09:48.763000 21018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:09:48.763078 21018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:09:48.763172 21017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:09:48.763968 21017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:09:48.769061 21017 catalog_manager.cc:1347] Generated new cluster ID: 77cb8ce1e84c4a3e98bc5b6f0e5f5be4
I20260924 07:09:48.769126 21017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:09:48.780099 21017 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:09:48.780925 21017 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:09:48.785604 21017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2: Generated new TSK 0
I20260924 07:09:48.786162 21017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
I20260924 07:09:48.814913 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:34005
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:46213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:48.920506 21024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:48.920727 21024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:48.920778 21024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:48.924662 21024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:48.924710 21024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:48.924769 21024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:09:48.929685 21024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:46213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:34005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:48.930526 21024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:48.931385 21024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:48.940294 21032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:48.940343 21029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:48.940343 21030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:48.940487 21024 server_base.cc:1034] running on GCE node
I20260924 07:09:48.940927 21024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:48.941504 21024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:48.942691 21024 hybrid_clock.cc:648] HybridClock initialized: now 1790233788942669 us; error 36 us; skew 500 ppm
Sep 24 07:09:48 dist-test-slave-2p7j krb5kdc[20917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233788, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:48.945259 21024 init.cc:377] Logged in from keytab as kudu/127.3.217.1@KRBTEST.COM (short username kudu)
I20260924 07:09:48.945996 21024 webserver.cc:460] Webserver started at http://127.3.217.1:35973/ using document root <none> and password file <none>
I20260924 07:09:48.946465 21024 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:48.946519 21024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:48.946677 21024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:48.948521 21024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "32c862c8b2e7476d82f088106535fa4f"
format_stamp: "Formatted at 2026-09-24 07:09:48 on dist-test-slave-2p7j"
server_key: "edd0f142c4d18a593572c4dd50f91bd5"
server_key_iv: "418cd9c21ff40a667856a9267590712e"
server_key_version: "encryptionkey@0"
I20260924 07:09:48.949002 21024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "32c862c8b2e7476d82f088106535fa4f"
format_stamp: "Formatted at 2026-09-24 07:09:48 on dist-test-slave-2p7j"
server_key: "edd0f142c4d18a593572c4dd50f91bd5"
server_key_iv: "418cd9c21ff40a667856a9267590712e"
server_key_version: "encryptionkey@0"
I20260924 07:09:48.952304 21024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:09:48.954895 21039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:48.956385 21024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:09:48.956487 21024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "32c862c8b2e7476d82f088106535fa4f"
format_stamp: "Formatted at 2026-09-24 07:09:48 on dist-test-slave-2p7j"
server_key: "edd0f142c4d18a593572c4dd50f91bd5"
server_key_iv: "418cd9c21ff40a667856a9267590712e"
server_key_version: "encryptionkey@0"
I20260924 07:09:48.956584 21024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:48.983938 21024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:48.986789 21024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:48.986951 21024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:48.987520 21024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:48.988292 21024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:48.988341 21024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:48.988379 21024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:48.988395 21024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:48.996991 21024 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:43329
I20260924 07:09:48.997081 21152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:43329 every 8 connection(s)
I20260924 07:09:48.997809 21024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:09:49.000864 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 21024
I20260924 07:09:49.000988 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:09:49.001284 3940 external_mini_cluster.cc:1428] Setting key c7fadb68eefba0731f58eef77ad331ff
Sep 24 07:09:49 dist-test-slave-2p7j krb5kdc[20917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233788, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.2@KRBTEST.COM" created.
I20260924 07:09:49.013154 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:48.999832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:53775 (local address 127.3.217.62:34005)
0924 07:09:48.999947 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:48.999950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:49.000982 (+ 1032us) server_negotiation.cc:407] Connection header received
0924 07:09:49.002401 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:49.002406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:49.002574 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:49.002691 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:49.003841 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.004527 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.006037 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.006288 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.006367 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:49.009058 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:49.009062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:49.009065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:49.009092 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:49.010702 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:49.011578 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:49.011582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:49.011583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:49.011690 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:49.012054 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:49.012057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:49.012059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:49.012233 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:49.012352 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:49.012804 (+ 452us) server_negotiation.cc:299] Negotiation successful
0924 07:09:49.012982 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 07:09:49.013823 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:49.000321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:34005 (local address 127.3.217.1:53775)
0924 07:09:49.000771 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:49.000818 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:49.002101 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:49.002894 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:49.002900 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:49.003245 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:49.003708 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:49.003721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.004650 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.004653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:49.005912 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:49.005924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.006431 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.006435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:49.006631 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:49.007207 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:49.007226 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:49.008899 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:49.010817 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:49.010821 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:49.010832 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:49.011348 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:49.011803 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:49.011806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:49.011808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:49.011954 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:49.012369 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:49.012375 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:49.012589 (+ 214us) client_negotiation.cc:769] Sending connection context
0924 07:09:49.012747 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 07:09:49.013057 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":108,"threads_started":1}
I20260924 07:09:49.015306 21153 heartbeater.cc:347] Connected to a master server at 127.3.217.62:34005
I20260924 07:09:49.015563 21153 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:49.016103 21153 heartbeater.cc:511] Master 127.3.217.62:34005 requested a full tablet report, sending...
I20260924 07:09:49.017797 20962 ts_manager.cc:194] Registered new tserver with Master: 32c862c8b2e7476d82f088106535fa4f (127.3.217.1:43329)
I20260924 07:09:49.019311 20962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.1@KRBTEST.COM'} at 127.3.217.1:53775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
I20260924 07:09:49.029069 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:34005
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:46213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:49.136852 21158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:49.137084 21158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:49.137125 21158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:49.141039 21158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:49.141088 21158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:49.141144 21158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:09:49.147025 21158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:46213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:34005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:49.148072 21158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:49.149161 21158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:49.158082 21166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:49.158080 21164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:49.158037 21163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:49.158339 21158 server_base.cc:1034] running on GCE node
I20260924 07:09:49.158685 21158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:49.159344 21158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:49.160630 21158 hybrid_clock.cc:648] HybridClock initialized: now 1790233789160592 us; error 36 us; skew 500 ppm
Sep 24 07:09:49 dist-test-slave-2p7j krb5kdc[20917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233789, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:49.164149 21158 init.cc:377] Logged in from keytab as kudu/127.3.217.2@KRBTEST.COM (short username kudu)
I20260924 07:09:49.165588 21158 webserver.cc:460] Webserver started at http://127.3.217.2:41187/ using document root <none> and password file <none>
I20260924 07:09:49.166157 21158 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:49.166224 21158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:49.166414 21158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:49.168382 21158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "3e5252f91ab24e74a88d020978737162"
format_stamp: "Formatted at 2026-09-24 07:09:49 on dist-test-slave-2p7j"
server_key: "f96e30c7cc1ad115958b44f80969f3b8"
server_key_iv: "025ad7258f7f0d57630c75770d70af19"
server_key_version: "encryptionkey@0"
I20260924 07:09:49.168917 21158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "3e5252f91ab24e74a88d020978737162"
format_stamp: "Formatted at 2026-09-24 07:09:49 on dist-test-slave-2p7j"
server_key: "f96e30c7cc1ad115958b44f80969f3b8"
server_key_iv: "025ad7258f7f0d57630c75770d70af19"
server_key_version: "encryptionkey@0"
I20260924 07:09:49.172328 21158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:09:49.174614 21173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:49.175949 21158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:09:49.176056 21158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "3e5252f91ab24e74a88d020978737162"
format_stamp: "Formatted at 2026-09-24 07:09:49 on dist-test-slave-2p7j"
server_key: "f96e30c7cc1ad115958b44f80969f3b8"
server_key_iv: "025ad7258f7f0d57630c75770d70af19"
server_key_version: "encryptionkey@0"
I20260924 07:09:49.176317 21158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:49.210420 21158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:49.213308 21158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:49.213462 21158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:49.213958 21158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:49.214740 21158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:49.214783 21158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:49.214823 21158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:49.214851 21158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:49.223289 21158 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:33367
I20260924 07:09:49.223330 21286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:33367 every 8 connection(s)
I20260924 07:09:49.224157 21158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:09:49.224617 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 21158
I20260924 07:09:49.224771 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:09:49.225191 3940 external_mini_cluster.cc:1428] Setting key d3441aede630fb3fbfa16ed22343d992
Sep 24 07:09:49 dist-test-slave-2p7j krb5kdc[20917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233789, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.3@KRBTEST.COM" created.
I20260924 07:09:49.238440 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:49.225830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:59467 (local address 127.3.217.62:34005)
0924 07:09:49.225953 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:49.225957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:49.226766 (+ 809us) server_negotiation.cc:407] Connection header received
0924 07:09:49.228241 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:49.228244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:49.228298 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:49.228368 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:49.229517 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.230051 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.231467 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.231848 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.231970 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:49.234231 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:49.234234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:49.234236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:49.234262 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:49.235595 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:49.236302 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:49.236305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:49.236306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:49.236406 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:49.236927 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:49.236929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:49.236930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:49.237103 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:49.237187 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:49.237964 (+ 777us) server_negotiation.cc:299] Negotiation successful
0924 07:09:49.238200 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:09:49.238914 21289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:49.226180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:34005 (local address 127.3.217.2:59467)
0924 07:09:49.226615 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:49.226673 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:49.228045 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:49.228527 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:49.228533 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:49.228853 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:49.229372 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:49.229389 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.230164 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.230168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:49.231330 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:49.231339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.231994 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.231997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:49.232131 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:49.232637 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:49.232653 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:49.234081 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:49.235820 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:49.235828 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:49.235845 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:49.236209 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:49.236522 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:49.236524 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:49.236525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:49.236808 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:49.237221 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:49.237227 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:49.237500 (+ 273us) client_negotiation.cc:769] Sending connection context
0924 07:09:49.237701 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 07:09:49.238001 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":102,"threads_started":1}
I20260924 07:09:49.240342 21287 heartbeater.cc:347] Connected to a master server at 127.3.217.62:34005
I20260924 07:09:49.240576 21287 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:49.241079 21287 heartbeater.cc:511] Master 127.3.217.62:34005 requested a full tablet report, sending...
I20260924 07:09:49.242161 20962 ts_manager.cc:194] Registered new tserver with Master: 3e5252f91ab24e74a88d020978737162 (127.3.217.2:33367)
I20260924 07:09:49.242916 20962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.2@KRBTEST.COM'} at 127.3.217.2:59467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
I20260924 07:09:49.255348 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:34005
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:46213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:49.360435 21292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:49.360656 21292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:49.360694 21292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:49.364580 21292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:49.364630 21292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:49.364684 21292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:09:49.369542 21292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:46213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:34005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:49.370401 21292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:49.371249 21292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:49.379973 21300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:49.380142 21292 server_base.cc:1034] running on GCE node
W20260924 07:09:49.379988 21297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:49.380020 21298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:49.380715 21292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:49.381271 21292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:49.382447 21292 hybrid_clock.cc:648] HybridClock initialized: now 1790233789382423 us; error 33 us; skew 500 ppm
Sep 24 07:09:49 dist-test-slave-2p7j krb5kdc[20917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233789, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:49.385064 21292 init.cc:377] Logged in from keytab as kudu/127.3.217.3@KRBTEST.COM (short username kudu)
I20260924 07:09:49.385857 21292 webserver.cc:460] Webserver started at http://127.3.217.3:42295/ using document root <none> and password file <none>
I20260924 07:09:49.386356 21292 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:49.386420 21292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:49.386605 21292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:49.388504 21292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "55ebcab9aceb4f67be8f6a15336e468c"
format_stamp: "Formatted at 2026-09-24 07:09:49 on dist-test-slave-2p7j"
server_key: "a6f0457ddd39ce0e9161636be574c31f"
server_key_iv: "f59a5e87660a39bf91e19e0736894e1c"
server_key_version: "encryptionkey@0"
I20260924 07:09:49.389010 21292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "55ebcab9aceb4f67be8f6a15336e468c"
format_stamp: "Formatted at 2026-09-24 07:09:49 on dist-test-slave-2p7j"
server_key: "a6f0457ddd39ce0e9161636be574c31f"
server_key_iv: "f59a5e87660a39bf91e19e0736894e1c"
server_key_version: "encryptionkey@0"
I20260924 07:09:49.392827 21292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:09:49.395675 21307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:49.396840 21292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:09:49.396948 21292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "55ebcab9aceb4f67be8f6a15336e468c"
format_stamp: "Formatted at 2026-09-24 07:09:49 on dist-test-slave-2p7j"
server_key: "a6f0457ddd39ce0e9161636be574c31f"
server_key_iv: "f59a5e87660a39bf91e19e0736894e1c"
server_key_version: "encryptionkey@0"
I20260924 07:09:49.397039 21292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:49.425467 21292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:49.428488 21292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:49.428686 21292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:49.429287 21292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:49.430341 21292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:49.430501 21292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:49.430584 21292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:49.430614 21292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:49.441211 21292 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:38179
I20260924 07:09:49.441283 21420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:38179 every 8 connection(s)
I20260924 07:09:49.442147 21292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
Sep 24 07:09:49 dist-test-slave-2p7j krb5kdc[20917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233789, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:09:49.451550 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 21292
I20260924 07:09:49.451664 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:09:49.451930 3940 external_mini_cluster.cc:1428] Setting key 8cda6f57f713e424bb4b4941cf5ee935
I20260924 07:09:49.455382 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:49.443947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:34781 (local address 127.3.217.62:34005)
0924 07:09:49.444084 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:49.444088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:49.444880 (+ 792us) server_negotiation.cc:407] Connection header received
0924 07:09:49.446239 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:49.446242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:49.446322 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:49.446394 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:49.447468 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.448070 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.449136 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.449366 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.449449 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:49.452022 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:49.452025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:49.452028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:49.452051 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:49.453467 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:49.454011 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:49.454014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:49.454015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:49.454087 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:49.454426 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:49.454429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:49.454430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:49.454589 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:49.454663 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:49.455163 (+ 500us) server_negotiation.cc:299] Negotiation successful
0924 07:09:49.455258 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 07:09:49.456175 21423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:49.444277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:34005 (local address 127.3.217.3:34781)
0924 07:09:49.444740 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:49.444780 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:49.446028 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:49.446560 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:49.446566 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:49.446902 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:49.447335 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:49.447348 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.448185 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.448187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:49.449028 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:49.449037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.449491 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.449495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:49.449648 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:49.450247 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:49.450265 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:49.451867 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:49.453570 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:49.453575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:49.453587 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:49.453932 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:49.454173 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:49.454175 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:49.454177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:49.454337 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:49.454720 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:49.454725 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:49.454934 (+ 209us) client_negotiation.cc:769] Sending connection context
0924 07:09:49.455098 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 07:09:49.455338 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":104,"threads_started":1}
I20260924 07:09:49.457139 21421 heartbeater.cc:347] Connected to a master server at 127.3.217.62:34005
I20260924 07:09:49.457325 21421 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:49.457708 21421 heartbeater.cc:511] Master 127.3.217.62:34005 requested a full tablet report, sending...
I20260924 07:09:49.458593 20962 ts_manager.cc:194] Registered new tserver with Master: 55ebcab9aceb4f67be8f6a15336e468c (127.3.217.3:38179)
I20260924 07:09:49.459221 20962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.3@KRBTEST.COM'} at 127.3.217.3:34781
I20260924 07:09:49.465615 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:09:49.474177 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:49.467445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41156 (local address 127.3.217.62:34005)
0924 07:09:49.467573 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:49.467577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:49.467756 (+ 179us) server_negotiation.cc:407] Connection header received
0924 07:09:49.467887 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:49.467890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:49.467927 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:49.467985 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:49.468639 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.469183 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.470002 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.470221 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.470299 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:49.471431 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:49.471435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:49.471437 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:49.471460 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:49.472749 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:49.473094 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:49.473097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:49.473099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:49.473176 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:49.473397 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:49.473400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:49.473401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:49.473588 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:49.473699 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:49.473915 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 07:09:49.474028 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:09:49.476554 20962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:09:49.479362 20962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:09:49.493535 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:49.487331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58930 (local address 127.3.217.1:43329)
0924 07:09:49.488663 (+ 1332us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:49.488679 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:49.488702 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:09:49.488769 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:49.488774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:49.488948 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:49.489097 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:49.489678 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.490921 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.492106 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.492887 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.492996 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:49.493141 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:49.493295 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 07:09:49.493386 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1149,"thread_start_us":167,"threads_started":1}
I20260924 07:09:49.493793 21436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:49.487805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59666 (local address 127.3.217.3:38179)
0924 07:09:49.488546 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:49.488556 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:49.488574 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:09:49.488630 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:49.488633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:49.488771 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:49.488900 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:49.489159 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.490361 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.492311 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.492939 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.493038 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:49.493163 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:49.493574 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 07:09:49.493660 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":131,"threads_started":1}
I20260924 07:09:49.493983 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:49.487166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:33367 (local address 127.0.0.1:53820)
0924 07:09:49.487994 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:49.488006 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:49.488087 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:49.489089 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:49.489093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:49.489116 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:49.489209 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:49.489215 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.490234 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.490238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:49.492135 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:49.492147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.493171 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.493174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:49.493273 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:49.493353 (+ 80us) client_negotiation.cc:769] Sending connection context
0924 07:09:49.493410 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 07:09:49.493490 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":112,"spinlock_wait_cycles":896,"thread_start_us":258,"threads_started":1}
I20260924 07:09:49.494012 21437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:49.488155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53820 (local address 127.3.217.2:33367)
0924 07:09:49.488632 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:49.488652 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:49.488680 (+ 28us) server_negotiation.cc:407] Connection header received
0924 07:09:49.488761 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:49.488767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:49.488948 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:49.489097 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:49.489300 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.490105 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.492280 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.493046 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.493536 (+ 490us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:49.493669 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:49.493716 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 07:09:49.493799 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":114,"threads_started":1}
I20260924 07:09:49.494022 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:49.487242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:38179 (local address 127.0.0.1:59666)
0924 07:09:49.488282 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:49.488292 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:49.488365 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:49.488906 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:49.488912 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:49.488933 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:49.489041 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:49.489048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.490498 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.490502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:49.492135 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:49.492147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.493221 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.493224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:49.493360 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:49.493380 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 07:09:49.493442 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 07:09:49.493508 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"mutex_wait_us":8,"thread_start_us":122,"threads_started":1}
I20260924 07:09:49.493508 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:49.487081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:43329 (local address 127.0.0.1:58930)
0924 07:09:49.487922 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:49.487968 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:49.488087 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:49.489285 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:49.489289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:49.489308 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:49.489425 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:49.489433 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.491068 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.491071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:49.491990 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:49.492000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.493007 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.493011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:49.493121 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:49.493156 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 07:09:49.493258 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 07:09:49.493371 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":1,"thread_start_us":66,"threads_started":1}
I20260924 07:09:49.496655 21221 tablet_service.cc:1467] Processing CreateTablet for tablet 38f21ca32a7b47548309383a55ae92f9 (DEFAULT_TABLE table=test-table [id=14d81b35a5184296a76ed26efa110765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:49.496814 21355 tablet_service.cc:1467] Processing CreateTablet for tablet 38f21ca32a7b47548309383a55ae92f9 (DEFAULT_TABLE table=test-table [id=14d81b35a5184296a76ed26efa110765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:49.496814 21086 tablet_service.cc:1467] Processing CreateTablet for tablet 38f21ca32a7b47548309383a55ae92f9 (DEFAULT_TABLE table=test-table [id=14d81b35a5184296a76ed26efa110765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:49.497545 21221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38f21ca32a7b47548309383a55ae92f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:49.497737 21355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38f21ca32a7b47548309383a55ae92f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:49.497737 21086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38f21ca32a7b47548309383a55ae92f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:49.503257 21438 tablet_bootstrap.cc:492] T 38f21ca32a7b47548309383a55ae92f9 P 3e5252f91ab24e74a88d020978737162: Bootstrap starting.
I20260924 07:09:49.504778 21439 tablet_bootstrap.cc:492] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f: Bootstrap starting.
I20260924 07:09:49.506269 21439 tablet_bootstrap.cc:654] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:49.506273 21438 tablet_bootstrap.cc:654] T 38f21ca32a7b47548309383a55ae92f9 P 3e5252f91ab24e74a88d020978737162: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:49.506928 21438 log.cc:824] T 38f21ca32a7b47548309383a55ae92f9 P 3e5252f91ab24e74a88d020978737162: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:49.507139 21439 log.cc:824] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:49.507555 21440 tablet_bootstrap.cc:492] T 38f21ca32a7b47548309383a55ae92f9 P 55ebcab9aceb4f67be8f6a15336e468c: Bootstrap starting.
I20260924 07:09:49.508760 21438 tablet_bootstrap.cc:492] T 38f21ca32a7b47548309383a55ae92f9 P 3e5252f91ab24e74a88d020978737162: No bootstrap required, opened a new log
I20260924 07:09:49.508940 21438 ts_tablet_manager.cc:1397] T 38f21ca32a7b47548309383a55ae92f9 P 3e5252f91ab24e74a88d020978737162: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260924 07:09:49.509101 21439 tablet_bootstrap.cc:492] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f: No bootstrap required, opened a new log
I20260924 07:09:49.509275 21439 ts_tablet_manager.cc:1397] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:09:49.509682 21440 tablet_bootstrap.cc:654] T 38f21ca32a7b47548309383a55ae92f9 P 55ebcab9aceb4f67be8f6a15336e468c: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:49.510756 21440 log.cc:824] T 38f21ca32a7b47548309383a55ae92f9 P 55ebcab9aceb4f67be8f6a15336e468c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:49.512558 21440 tablet_bootstrap.cc:492] T 38f21ca32a7b47548309383a55ae92f9 P 55ebcab9aceb4f67be8f6a15336e468c: No bootstrap required, opened a new log
I20260924 07:09:49.512722 21440 ts_tablet_manager.cc:1397] T 38f21ca32a7b47548309383a55ae92f9 P 55ebcab9aceb4f67be8f6a15336e468c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:09:49.512434 21438 raft_consensus.cc:348] T 38f21ca32a7b47548309383a55ae92f9 P 3e5252f91ab24e74a88d020978737162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5252f91ab24e74a88d020978737162" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 33367 } } peers { permanent_uuid: "32c862c8b2e7476d82f088106535fa4f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 43329 } } peers { permanent_uuid: "55ebcab9aceb4f67be8f6a15336e468c" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 38179 } }
I20260924 07:09:49.512732 21439 raft_consensus.cc:348] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5252f91ab24e74a88d020978737162" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 33367 } } peers { permanent_uuid: "32c862c8b2e7476d82f088106535fa4f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 43329 } } peers { permanent_uuid: "55ebcab9aceb4f67be8f6a15336e468c" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 38179 } }
I20260924 07:09:49.512914 21438 raft_consensus.cc:374] T 38f21ca32a7b47548309383a55ae92f9 P 3e5252f91ab24e74a88d020978737162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:49.512985 21438 raft_consensus.cc:729] T 38f21ca32a7b47548309383a55ae92f9 P 3e5252f91ab24e74a88d020978737162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e5252f91ab24e74a88d020978737162, State: Initialized, Role: FOLLOWER
I20260924 07:09:49.512996 21439 raft_consensus.cc:374] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:49.513057 21439 raft_consensus.cc:729] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c862c8b2e7476d82f088106535fa4f, State: Initialized, Role: FOLLOWER
I20260924 07:09:49.513528 21438 consensus_queue.cc:260] T 38f21ca32a7b47548309383a55ae92f9 P 3e5252f91ab24e74a88d020978737162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5252f91ab24e74a88d020978737162" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 33367 } } peers { permanent_uuid: "32c862c8b2e7476d82f088106535fa4f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 43329 } } peers { permanent_uuid: "55ebcab9aceb4f67be8f6a15336e468c" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 38179 } }
I20260924 07:09:49.513550 21439 consensus_queue.cc:260] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5252f91ab24e74a88d020978737162" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 33367 } } peers { permanent_uuid: "32c862c8b2e7476d82f088106535fa4f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 43329 } } peers { permanent_uuid: "55ebcab9aceb4f67be8f6a15336e468c" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 38179 } }
I20260924 07:09:49.514359 21287 heartbeater.cc:503] Master 127.3.217.62:34005 was elected leader, sending a full tablet report...
I20260924 07:09:49.514477 21439 ts_tablet_manager.cc:1428] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:09:49.514731 21438 ts_tablet_manager.cc:1428] T 38f21ca32a7b47548309383a55ae92f9 P 3e5252f91ab24e74a88d020978737162: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 07:09:49.514833 21153 heartbeater.cc:503] Master 127.3.217.62:34005 was elected leader, sending a full tablet report...
I20260924 07:09:49.516171 21440 raft_consensus.cc:348] T 38f21ca32a7b47548309383a55ae92f9 P 55ebcab9aceb4f67be8f6a15336e468c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5252f91ab24e74a88d020978737162" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 33367 } } peers { permanent_uuid: "32c862c8b2e7476d82f088106535fa4f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 43329 } } peers { permanent_uuid: "55ebcab9aceb4f67be8f6a15336e468c" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 38179 } }
I20260924 07:09:49.516445 21440 raft_consensus.cc:374] T 38f21ca32a7b47548309383a55ae92f9 P 55ebcab9aceb4f67be8f6a15336e468c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:49.516506 21440 raft_consensus.cc:729] T 38f21ca32a7b47548309383a55ae92f9 P 55ebcab9aceb4f67be8f6a15336e468c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55ebcab9aceb4f67be8f6a15336e468c, State: Initialized, Role: FOLLOWER
I20260924 07:09:49.516911 21440 consensus_queue.cc:260] T 38f21ca32a7b47548309383a55ae92f9 P 55ebcab9aceb4f67be8f6a15336e468c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5252f91ab24e74a88d020978737162" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 33367 } } peers { permanent_uuid: "32c862c8b2e7476d82f088106535fa4f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 43329 } } peers { permanent_uuid: "55ebcab9aceb4f67be8f6a15336e468c" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 38179 } }
I20260924 07:09:49.517803 21440 ts_tablet_manager.cc:1428] T 38f21ca32a7b47548309383a55ae92f9 P 55ebcab9aceb4f67be8f6a15336e468c: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260924 07:09:49.518187 21421 heartbeater.cc:503] Master 127.3.217.62:34005 was elected leader, sending a full tablet report...
W20260924 07:09:49.693259 21422 tablet.cc:2369] T 38f21ca32a7b47548309383a55ae92f9 P 55ebcab9aceb4f67be8f6a15336e468c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:09:49.703184 21444 raft_consensus.cc:482] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:09:49.703411 21444 raft_consensus.cc:504] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5252f91ab24e74a88d020978737162" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 33367 } } peers { permanent_uuid: "32c862c8b2e7476d82f088106535fa4f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 43329 } } peers { permanent_uuid: "55ebcab9aceb4f67be8f6a15336e468c" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 38179 } }
I20260924 07:09:49.704617 21444 leader_election.cc:288] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e5252f91ab24e74a88d020978737162 (127.3.217.2:33367), 55ebcab9aceb4f67be8f6a15336e468c (127.3.217.3:38179)
I20260924 07:09:49.709945 21436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:49.704961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:54597 (local address 127.3.217.3:38179)
0924 07:09:49.705181 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:49.705185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:49.705334 (+ 149us) server_negotiation.cc:407] Connection header received
0924 07:09:49.705650 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:49.705653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:49.705706 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:49.705795 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:49.706137 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.706916 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.708644 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.709248 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.709355 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:49.709414 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:49.709746 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 07:09:49.709823 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260924 07:09:49.709945 21437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:49.705022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:44767 (local address 127.3.217.2:33367)
0924 07:09:49.705167 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:49.705172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:49.705334 (+ 162us) server_negotiation.cc:407] Connection header received
0924 07:09:49.705650 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:49.705653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:49.705734 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:49.705807 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:49.706143 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.706940 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.708644 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.709248 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.709355 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:49.709414 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:49.709746 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 07:09:49.709823 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:09:49.710000 21448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:49.704860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:33367 (local address 127.3.217.1:44767)
0924 07:09:49.705232 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:49.705243 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:49.705361 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:49.705820 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:49.705823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:49.705852 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:49.706033 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:49.706038 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.707056 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.707060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:49.708517 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:49.708527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.709403 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.709407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:49.709583 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:49.709626 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 07:09:49.709713 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 07:09:49.709832 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":234,"spinlock_wait_cycles":13312,"thread_start_us":135,"threads_started":1}
I20260924 07:09:49.710166 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:49.704858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:38179 (local address 127.3.217.1:54597)
0924 07:09:49.705221 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:49.705236 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:49.705371 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:49.705820 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:49.705824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:49.705852 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:49.706026 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:49.706032 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.707056 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.707060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:49.708519 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:49.708527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.709403 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.709407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:49.709563 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:49.709621 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 07:09:49.709717 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 07:09:49.709825 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":184,"spinlock_wait_cycles":23552,"thread_start_us":113,"threads_started":1}
I20260924 07:09:49.710577 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38f21ca32a7b47548309383a55ae92f9" candidate_uuid: "32c862c8b2e7476d82f088106535fa4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e5252f91ab24e74a88d020978737162" is_pre_election: true
I20260924 07:09:49.710765 21375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38f21ca32a7b47548309383a55ae92f9" candidate_uuid: "32c862c8b2e7476d82f088106535fa4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55ebcab9aceb4f67be8f6a15336e468c" is_pre_election: true
I20260924 07:09:49.710851 21241 raft_consensus.cc:2454] T 38f21ca32a7b47548309383a55ae92f9 P 3e5252f91ab24e74a88d020978737162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32c862c8b2e7476d82f088106535fa4f in term 0.
I20260924 07:09:49.710996 21375 raft_consensus.cc:2454] T 38f21ca32a7b47548309383a55ae92f9 P 55ebcab9aceb4f67be8f6a15336e468c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32c862c8b2e7476d82f088106535fa4f in term 0.
I20260924 07:09:49.711277 21043 leader_election.cc:302] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32c862c8b2e7476d82f088106535fa4f, 3e5252f91ab24e74a88d020978737162; no voters:
I20260924 07:09:49.711551 21444 raft_consensus.cc:2781] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:09:49.711628 21444 raft_consensus.cc:482] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:09:49.711673 21444 raft_consensus.cc:3037] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:49.712628 21444 raft_consensus.cc:504] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5252f91ab24e74a88d020978737162" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 33367 } } peers { permanent_uuid: "32c862c8b2e7476d82f088106535fa4f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 43329 } } peers { permanent_uuid: "55ebcab9aceb4f67be8f6a15336e468c" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 38179 } }
I20260924 07:09:49.712944 21444 leader_election.cc:288] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f [CANDIDATE]: Term 1 election: Requested vote from peers 3e5252f91ab24e74a88d020978737162 (127.3.217.2:33367), 55ebcab9aceb4f67be8f6a15336e468c (127.3.217.3:38179)
I20260924 07:09:49.713241 21241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38f21ca32a7b47548309383a55ae92f9" candidate_uuid: "32c862c8b2e7476d82f088106535fa4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e5252f91ab24e74a88d020978737162"
I20260924 07:09:49.713264 21375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38f21ca32a7b47548309383a55ae92f9" candidate_uuid: "32c862c8b2e7476d82f088106535fa4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55ebcab9aceb4f67be8f6a15336e468c"
I20260924 07:09:49.713335 21241 raft_consensus.cc:3037] T 38f21ca32a7b47548309383a55ae92f9 P 3e5252f91ab24e74a88d020978737162 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:49.713388 21375 raft_consensus.cc:3037] T 38f21ca32a7b47548309383a55ae92f9 P 55ebcab9aceb4f67be8f6a15336e468c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:49.714267 21241 raft_consensus.cc:2454] T 38f21ca32a7b47548309383a55ae92f9 P 3e5252f91ab24e74a88d020978737162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32c862c8b2e7476d82f088106535fa4f in term 1.
I20260924 07:09:49.714355 21375 raft_consensus.cc:2454] T 38f21ca32a7b47548309383a55ae92f9 P 55ebcab9aceb4f67be8f6a15336e468c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32c862c8b2e7476d82f088106535fa4f in term 1.
I20260924 07:09:49.714507 21043 leader_election.cc:302] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32c862c8b2e7476d82f088106535fa4f, 3e5252f91ab24e74a88d020978737162; no voters:
I20260924 07:09:49.714651 21444 raft_consensus.cc:2781] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:49.714841 21444 raft_consensus.cc:686] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f [term 1 LEADER]: Becoming Leader. State: Replica: 32c862c8b2e7476d82f088106535fa4f, State: Running, Role: LEADER
I20260924 07:09:49.715133 21444 consensus_queue.cc:237] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5252f91ab24e74a88d020978737162" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 33367 } } peers { permanent_uuid: "32c862c8b2e7476d82f088106535fa4f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 43329 } } peers { permanent_uuid: "55ebcab9aceb4f67be8f6a15336e468c" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 38179 } }
I20260924 07:09:49.718815 20960 catalog_manager.cc:5495] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 32c862c8b2e7476d82f088106535fa4f (127.3.217.1). New cstate: current_term: 1 leader_uuid: "32c862c8b2e7476d82f088106535fa4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e5252f91ab24e74a88d020978737162" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 33367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32c862c8b2e7476d82f088106535fa4f" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 43329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55ebcab9aceb4f67be8f6a15336e468c" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 38179 } health_report { overall_health: UNKNOWN } } }
W20260924 07:09:49.725509 21288 tablet.cc:2369] T 38f21ca32a7b47548309383a55ae92f9 P 3e5252f91ab24e74a88d020978737162: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:09:49.749025 21154 tablet.cc:2369] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:09:49.784814 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:49.781573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58934 (local address 127.3.217.1:43329)
0924 07:09:49.781705 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:49.781709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:49.781719 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:09:49.781791 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:49.781793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:49.781842 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:49.781928 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:09:49.782170 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.782676 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.783487 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.783713 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.783802 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:49.784065 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:09:49.784460 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:09:49.784561 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:49.784632 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 07:09:49.784684 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:09:49.790184 21375 raft_consensus.cc:1261] T 38f21ca32a7b47548309383a55ae92f9 P 55ebcab9aceb4f67be8f6a15336e468c [term 1 FOLLOWER]: Refusing update from remote peer 32c862c8b2e7476d82f088106535fa4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:49.790197 21241 raft_consensus.cc:1261] T 38f21ca32a7b47548309383a55ae92f9 P 3e5252f91ab24e74a88d020978737162 [term 1 FOLLOWER]: Refusing update from remote peer 32c862c8b2e7476d82f088106535fa4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:49.790761 21449 consensus_queue.cc:1035] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "55ebcab9aceb4f67be8f6a15336e468c" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 38179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:49.790949 21444 consensus_queue.cc:1035] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e5252f91ab24e74a88d020978737162" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 33367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:49.805752 20961 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:49.805888 20961 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:49.808035 20961 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02a4ed4653f24b79a0a70cf31e4902a2: Sending DeleteTablet for 3 replicas of tablet 38f21ca32a7b47548309383a55ae92f9
I20260924 07:09:49.808700 21221 tablet_service.cc:1514] Processing DeleteTablet for tablet 38f21ca32a7b47548309383a55ae92f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:49 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:53820
I20260924 07:09:49.808753 21086 tablet_service.cc:1514] Processing DeleteTablet for tablet 38f21ca32a7b47548309383a55ae92f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:49 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:58930
I20260924 07:09:49.808956 21355 tablet_service.cc:1514] Processing DeleteTablet for tablet 38f21ca32a7b47548309383a55ae92f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:49 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:59666
I20260924 07:09:49.809213 21463 tablet_replica.cc:330] T 38f21ca32a7b47548309383a55ae92f9 P 3e5252f91ab24e74a88d020978737162: stopping tablet replica
I20260924 07:09:49.809451 21462 tablet_replica.cc:330] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f: stopping tablet replica
I20260924 07:09:49.809734 21462 raft_consensus.cc:2229] T 38f21ca32a7b47548309383a55ae92f9 P 32c862c8b2e7476d82f088106535fa4f [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:09:49.809818 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 21024
I20260924 07:09:49.809818 21463 raft_consensus.cc:2229] T 38f21ca32a7b47548309383a55ae92f9 P 3e5252f91ab24e74a88d020978737162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:49.810711 21463 raft_consensus.cc:2258] T 38f21ca32a7b47548309383a55ae92f9 P 3e5252f91ab24e74a88d020978737162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:49.810758 21464 tablet_replica.cc:330] T 38f21ca32a7b47548309383a55ae92f9 P 55ebcab9aceb4f67be8f6a15336e468c: stopping tablet replica
I20260924 07:09:49.810959 21464 raft_consensus.cc:2229] T 38f21ca32a7b47548309383a55ae92f9 P 55ebcab9aceb4f67be8f6a15336e468c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:49.811161 21464 raft_consensus.cc:2258] T 38f21ca32a7b47548309383a55ae92f9 P 55ebcab9aceb4f67be8f6a15336e468c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:49.812077 21463 ts_tablet_manager.cc:1905] T 38f21ca32a7b47548309383a55ae92f9 P 3e5252f91ab24e74a88d020978737162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:49.814073 21463 ts_tablet_manager.cc:1918] T 38f21ca32a7b47548309383a55ae92f9 P 3e5252f91ab24e74a88d020978737162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:09:49.814147 21463 log.cc:1196] T 38f21ca32a7b47548309383a55ae92f9 P 3e5252f91ab24e74a88d020978737162: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/38f21ca32a7b47548309383a55ae92f9
I20260924 07:09:49.814395 21463 ts_tablet_manager.cc:1939] T 38f21ca32a7b47548309383a55ae92f9 P 3e5252f91ab24e74a88d020978737162: Deleting consensus metadata
W20260924 07:09:49.816039 20948 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.217.1:43329 (error 108)
I20260924 07:09:49.816195 20949 catalog_manager.cc:4841] TS 3e5252f91ab24e74a88d020978737162 (127.3.217.2:33367): tablet 38f21ca32a7b47548309383a55ae92f9 (table test-table [id=14d81b35a5184296a76ed26efa110765]) successfully deleted
I20260924 07:09:49.816263 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 21158
I20260924 07:09:49.818184 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:49.817940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:43329 (local address 127.0.0.1:58946)
0924 07:09:49.818060 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:49.818118 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.1:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 07:09:49.818374 21464 ts_tablet_manager.cc:1905] T 38f21ca32a7b47548309383a55ae92f9 P 55ebcab9aceb4f67be8f6a15336e468c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 07:09:49.819651 20948 catalog_manager.cc:4584] TS 32c862c8b2e7476d82f088106535fa4f (127.3.217.1:43329): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38f21ca32a7b47548309383a55ae92f9: Network error: Client connection negotiation failed: client connection to 127.3.217.1:43329: connect: Connection refused (error 111)
I20260924 07:09:49.820861 21464 ts_tablet_manager.cc:1918] T 38f21ca32a7b47548309383a55ae92f9 P 55ebcab9aceb4f67be8f6a15336e468c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:09:49.820950 21464 log.cc:1196] T 38f21ca32a7b47548309383a55ae92f9 P 55ebcab9aceb4f67be8f6a15336e468c: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/38f21ca32a7b47548309383a55ae92f9
I20260924 07:09:49.821219 21464 ts_tablet_manager.cc:1939] T 38f21ca32a7b47548309383a55ae92f9 P 55ebcab9aceb4f67be8f6a15336e468c: Deleting consensus metadata
I20260924 07:09:49.822516 20946 catalog_manager.cc:4841] TS 55ebcab9aceb4f67be8f6a15336e468c (127.3.217.3:38179): tablet 38f21ca32a7b47548309383a55ae92f9 (table test-table [id=14d81b35a5184296a76ed26efa110765]) successfully deleted
I20260924 07:09:49.822654 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 21292
I20260924 07:09:49.827039 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 20930
2026-09-24T07:09:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2344 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T07:09:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:09:49Z Disabled control of system clock
I20260924 07:09:49.856220 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:42751
--webserver_interface=127.3.217.62
--webserver_port=0
--builtin_ntp_servers=127.3.217.20:36001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:42751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:09:49.958292 21469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:49.958523 21469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:49.958556 21469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:49.962476 21469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:09:49.962525 21469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:49.962539 21469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:49.962554 21469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:09:49.962568 21469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:09:49.967176 21469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:36001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:42751
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:42751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:49.968138 21469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:49.969528 21469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:49.977324 21474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:49.977339 21477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:49.977483 21469 server_base.cc:1034] running on GCE node
W20260924 07:09:49.977669 21475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:49.978188 21469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:49.979163 21469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:49.980334 21469 hybrid_clock.cc:648] HybridClock initialized: now 1790233789980318 us; error 32 us; skew 500 ppm
I20260924 07:09:49.987815 21469 webserver.cc:460] Webserver started at http://127.3.217.62:38151/ using document root <none> and password file <none>
I20260924 07:09:49.988341 21469 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:49.988411 21469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:49.988698 21469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:49.990619 21469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "6c45e65f5b1345e8838e40008b29cc46"
format_stamp: "Formatted at 2026-09-24 07:09:49 on dist-test-slave-2p7j"
server_key: "e5223c81b856c0c607002f07ae74783a"
server_key_iv: "c449e8eca05ef87b495a172eddc75202"
server_key_version: "encryptionkey@0"
I20260924 07:09:49.991112 21469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "6c45e65f5b1345e8838e40008b29cc46"
format_stamp: "Formatted at 2026-09-24 07:09:49 on dist-test-slave-2p7j"
server_key: "e5223c81b856c0c607002f07ae74783a"
server_key_iv: "c449e8eca05ef87b495a172eddc75202"
server_key_version: "encryptionkey@0"
I20260924 07:09:49.994335 21469 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 07:09:49.996487 21483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:49.997641 21469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:09:49.997766 21469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "6c45e65f5b1345e8838e40008b29cc46"
format_stamp: "Formatted at 2026-09-24 07:09:49 on dist-test-slave-2p7j"
server_key: "e5223c81b856c0c607002f07ae74783a"
server_key_iv: "c449e8eca05ef87b495a172eddc75202"
server_key_version: "encryptionkey@0"
I20260924 07:09:49.997850 21469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:50.017611 21469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:50.018330 21469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:50.018498 21469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:50.025023 21469 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:42751
I20260924 07:09:50.025029 21535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:42751 every 8 connection(s)
I20260924 07:09:50.025906 21469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:09:50.028807 21536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:50.031527 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 21469
I20260924 07:09:50.031684 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:09:50.032173 3940 external_mini_cluster.cc:1428] Setting key cf0816ab927ceaec2d2a052d845e5210
I20260924 07:09:50.034157 21536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c45e65f5b1345e8838e40008b29cc46: Bootstrap starting.
I20260924 07:09:50.036273 21536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c45e65f5b1345e8838e40008b29cc46: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:50.036916 21536 log.cc:824] T 00000000000000000000000000000000 P 6c45e65f5b1345e8838e40008b29cc46: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:50.038537 21536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c45e65f5b1345e8838e40008b29cc46: No bootstrap required, opened a new log
I20260924 07:09:50.040831 21539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.033687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40790 (local address 127.3.217.62:42751)
0924 07:09:50.034274 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:50.034293 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:50.034328 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:09:50.034950 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:50.034966 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:50.035338 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:50.035653 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:50.035972 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.036922 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.037742 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.038213 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.038360 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.039132 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:50.039137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:50.039148 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:50.039529 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:50.039585 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:50.039605 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:50.039687 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:50.039813 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 07:09:50.040038 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":118,"threads_started":1}
I20260924 07:09:50.041473 21536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c45e65f5b1345e8838e40008b29cc46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c45e65f5b1345e8838e40008b29cc46" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42751 } }
I20260924 07:09:50.041714 21536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c45e65f5b1345e8838e40008b29cc46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:50.041767 21536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c45e65f5b1345e8838e40008b29cc46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c45e65f5b1345e8838e40008b29cc46, State: Initialized, Role: FOLLOWER
I20260924 07:09:50.042114 21536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c45e65f5b1345e8838e40008b29cc46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c45e65f5b1345e8838e40008b29cc46" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42751 } }
I20260924 07:09:50.042222 21536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c45e65f5b1345e8838e40008b29cc46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:09:50.042484 21536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c45e65f5b1345e8838e40008b29cc46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:09:50.042616 21536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c45e65f5b1345e8838e40008b29cc46 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:50.043886 21536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c45e65f5b1345e8838e40008b29cc46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c45e65f5b1345e8838e40008b29cc46" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42751 } }
I20260924 07:09:50.044209 21536 leader_election.cc:302] T 00000000000000000000000000000000 P 6c45e65f5b1345e8838e40008b29cc46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c45e65f5b1345e8838e40008b29cc46; no voters:
I20260924 07:09:50.044585 21536 leader_election.cc:288] T 00000000000000000000000000000000 P 6c45e65f5b1345e8838e40008b29cc46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:09:50.044727 21541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c45e65f5b1345e8838e40008b29cc46 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:50.044915 21541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c45e65f5b1345e8838e40008b29cc46 [term 1 LEADER]: Becoming Leader. State: Replica: 6c45e65f5b1345e8838e40008b29cc46, State: Running, Role: LEADER
I20260924 07:09:50.045238 21541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c45e65f5b1345e8838e40008b29cc46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c45e65f5b1345e8838e40008b29cc46" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42751 } }
I20260924 07:09:50.045677 21536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c45e65f5b1345e8838e40008b29cc46 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:09:50.046681 21542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c45e65f5b1345e8838e40008b29cc46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c45e65f5b1345e8838e40008b29cc46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c45e65f5b1345e8838e40008b29cc46" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42751 } } }
I20260924 07:09:50.046809 21542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c45e65f5b1345e8838e40008b29cc46 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:50.047158 21550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:09:50.047168 21543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c45e65f5b1345e8838e40008b29cc46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c45e65f5b1345e8838e40008b29cc46. Latest consensus state: current_term: 1 leader_uuid: "6c45e65f5b1345e8838e40008b29cc46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c45e65f5b1345e8838e40008b29cc46" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42751 } } }
I20260924 07:09:50.047874 21543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c45e65f5b1345e8838e40008b29cc46 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:50.050159 21550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:09:50.055251 21550 catalog_manager.cc:1347] Generated new cluster ID: d6f1369fd777458492ea0299b9dfb37e
I20260924 07:09:50.055322 21550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:09:50.071471 21550 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:09:50.072341 21550 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:09:50.078269 21550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c45e65f5b1345e8838e40008b29cc46: Generated new TSK 0
I20260924 07:09:50.078853 21550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:09:50.090210 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42751
--builtin_ntp_servers=127.3.217.20:36001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:09:50.222182 21560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:50.222417 21560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:50.222456 21560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:50.226576 21560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:50.226629 21560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:50.226682 21560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:09:50.231353 21560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:36001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:50.232268 21560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:50.233494 21560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:50.243453 21565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:50.243453 21568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:50.243453 21566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:50.243957 21560 server_base.cc:1034] running on GCE node
I20260924 07:09:50.244318 21560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:50.245004 21560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:50.246163 21560 hybrid_clock.cc:648] HybridClock initialized: now 1790233790246149 us; error 30 us; skew 500 ppm
I20260924 07:09:50.247956 21560 webserver.cc:460] Webserver started at http://127.3.217.1:46749/ using document root <none> and password file <none>
I20260924 07:09:50.248481 21560 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:50.248549 21560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:50.248731 21560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:50.251075 21560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "eaab69c78ead4929be5906b473bddb1a"
format_stamp: "Formatted at 2026-09-24 07:09:50 on dist-test-slave-2p7j"
server_key: "10c46c4696c4b005a2b189906f60240d"
server_key_iv: "73035f8e0242268a1f41f7a5374b707b"
server_key_version: "encryptionkey@0"
I20260924 07:09:50.251650 21560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "eaab69c78ead4929be5906b473bddb1a"
format_stamp: "Formatted at 2026-09-24 07:09:50 on dist-test-slave-2p7j"
server_key: "10c46c4696c4b005a2b189906f60240d"
server_key_iv: "73035f8e0242268a1f41f7a5374b707b"
server_key_version: "encryptionkey@0"
I20260924 07:09:50.254997 21560 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:09:50.257258 21574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:50.258355 21560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:09:50.258456 21560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "eaab69c78ead4929be5906b473bddb1a"
format_stamp: "Formatted at 2026-09-24 07:09:50 on dist-test-slave-2p7j"
server_key: "10c46c4696c4b005a2b189906f60240d"
server_key_iv: "73035f8e0242268a1f41f7a5374b707b"
server_key_version: "encryptionkey@0"
I20260924 07:09:50.258544 21560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:50.282335 21560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:50.283049 21560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:50.283253 21560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:50.283969 21560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:50.284945 21560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:50.285000 21560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:50.285058 21560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:50.285092 21560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:50.293738 21560 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:44075
I20260924 07:09:50.293752 21687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:44075 every 8 connection(s)
I20260924 07:09:50.294569 21560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:09:50.297389 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 21560
I20260924 07:09:50.297509 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:09:50.297829 3940 external_mini_cluster.cc:1428] Setting key 3aee466cbcee9a2f889ba3ba454a0e27
I20260924 07:09:50.299695 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42751
--builtin_ntp_servers=127.3.217.20:36001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 07:09:50.304319 21539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.296252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:47911 (local address 127.3.217.62:42751)
0924 07:09:50.296369 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:50.296373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:50.297575 (+ 1202us) server_negotiation.cc:407] Connection header received
0924 07:09:50.299171 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:50.299183 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:50.299231 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:50.299299 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:50.300399 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.300943 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.302392 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.302640 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.302746 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.303902 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:50.303905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:50.303908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:50.303920 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:50.303939 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:50.303947 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:50.304017 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:50.304164 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 07:09:50.304213 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:09:50.305433 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.296588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42751 (local address 127.3.217.1:47911)
0924 07:09:50.297426 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:50.297480 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:50.298956 (+ 1476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:50.299484 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:50.299493 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:50.299618 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:50.300250 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:50.300268 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.301091 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.301094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:50.302254 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:50.302265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.302777 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.302781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:50.302964 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.303602 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:09:50.303604 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:50.303659 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:09:50.303740 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:50.304029 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:50.304037 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:50.304043 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:09:50.304168 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 07:09:50.304467 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":140,"threads_started":1}
I20260924 07:09:50.306654 21688 heartbeater.cc:347] Connected to a master server at 127.3.217.62:42751
I20260924 07:09:50.306900 21688 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:50.307384 21688 heartbeater.cc:511] Master 127.3.217.62:42751 requested a full tablet report, sending...
I20260924 07:09:50.308810 21500 ts_manager.cc:194] Registered new tserver with Master: eaab69c78ead4929be5906b473bddb1a (127.3.217.1:44075)
I20260924 07:09:50.310216 21500 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.217.1:47911
W20260924 07:09:50.404999 21691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:50.405218 21691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:50.405267 21691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:50.409104 21691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:50.409147 21691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:50.409220 21691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:09:50.413692 21691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:36001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:50.414561 21691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:50.415763 21691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:50.425441 21699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:50.425428 21697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:50.425428 21696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:50.425871 21691 server_base.cc:1034] running on GCE node
I20260924 07:09:50.426249 21691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:50.426791 21691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:50.427944 21691 hybrid_clock.cc:648] HybridClock initialized: now 1790233790427928 us; error 32 us; skew 500 ppm
I20260924 07:09:50.429718 21691 webserver.cc:460] Webserver started at http://127.3.217.2:37335/ using document root <none> and password file <none>
I20260924 07:09:50.430224 21691 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:50.430282 21691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:50.430430 21691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:50.432353 21691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "c87442f63bb44237a5007e749930c184"
format_stamp: "Formatted at 2026-09-24 07:09:50 on dist-test-slave-2p7j"
server_key: "fbb0874efc4398a52b6d63d0b4db9cae"
server_key_iv: "67907c68df0fde5038875a69dd68de22"
server_key_version: "encryptionkey@0"
I20260924 07:09:50.432833 21691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "c87442f63bb44237a5007e749930c184"
format_stamp: "Formatted at 2026-09-24 07:09:50 on dist-test-slave-2p7j"
server_key: "fbb0874efc4398a52b6d63d0b4db9cae"
server_key_iv: "67907c68df0fde5038875a69dd68de22"
server_key_version: "encryptionkey@0"
I20260924 07:09:50.436120 21691 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:09:50.438450 21705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:50.439724 21691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:09:50.439828 21691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "c87442f63bb44237a5007e749930c184"
format_stamp: "Formatted at 2026-09-24 07:09:50 on dist-test-slave-2p7j"
server_key: "fbb0874efc4398a52b6d63d0b4db9cae"
server_key_iv: "67907c68df0fde5038875a69dd68de22"
server_key_version: "encryptionkey@0"
I20260924 07:09:50.439919 21691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:50.451656 21691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:50.452234 21691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:50.452390 21691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:50.452893 21691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:50.453975 21691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:50.454041 21691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:50.454099 21691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:50.454133 21691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:50.463032 21691 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:42175
I20260924 07:09:50.463074 21818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:42175 every 8 connection(s)
I20260924 07:09:50.463889 21691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:09:50.465953 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 21691
I20260924 07:09:50.466063 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:09:50.466377 3940 external_mini_cluster.cc:1428] Setting key d19aad64d669b28f014749fa9ef1b684
I20260924 07:09:50.468354 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42751
--builtin_ntp_servers=127.3.217.20:36001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 07:09:50.473469 21539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.465560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:55523 (local address 127.3.217.62:42751)
0924 07:09:50.465719 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:50.465723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:50.466773 (+ 1050us) server_negotiation.cc:407] Connection header received
0924 07:09:50.468435 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:50.468438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:50.468490 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:50.468563 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:50.469718 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.470522 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.471655 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.471972 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.472073 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.473022 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:50.473026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:50.473029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:50.473043 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:50.473064 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:50.473073 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:50.473150 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:50.473267 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 07:09:50.473326 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 07:09:50.474817 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.465945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42751 (local address 127.3.217.2:55523)
0924 07:09:50.466640 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:50.466681 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:50.468236 (+ 1555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:50.468856 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:50.468864 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:50.468960 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:50.469549 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:50.469565 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.470656 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.470660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:50.471542 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:50.471553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.472119 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.472122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:50.472258 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.472790 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:09:50.472792 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:50.472827 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:09:50.472883 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:50.473149 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:50.473155 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:50.473158 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:09:50.473242 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 07:09:50.473434 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":130,"threads_started":1}
I20260924 07:09:50.475873 21819 heartbeater.cc:347] Connected to a master server at 127.3.217.62:42751
I20260924 07:09:50.476071 21819 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:50.476516 21819 heartbeater.cc:511] Master 127.3.217.62:42751 requested a full tablet report, sending...
I20260924 07:09:50.477445 21500 ts_manager.cc:194] Registered new tserver with Master: c87442f63bb44237a5007e749930c184 (127.3.217.2:42175)
I20260924 07:09:50.478169 21500 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.217.2:55523
W20260924 07:09:50.579852 21822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:50.580070 21822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:50.580104 21822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:50.583931 21822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:50.583974 21822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:50.584024 21822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:09:50.588534 21822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:36001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:50.589418 21822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:50.590610 21822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:50.599638 21827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:50.599709 21828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:50.599990 21822 server_base.cc:1034] running on GCE node
W20260924 07:09:50.599749 21830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:50.600473 21822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:50.601007 21822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:50.602180 21822 hybrid_clock.cc:648] HybridClock initialized: now 1790233790602168 us; error 28 us; skew 500 ppm
I20260924 07:09:50.603922 21822 webserver.cc:460] Webserver started at http://127.3.217.3:40221/ using document root <none> and password file <none>
I20260924 07:09:50.604506 21822 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:50.604573 21822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:50.604741 21822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:50.606596 21822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "7fd533bd76f6421b94967a2d54012146"
format_stamp: "Formatted at 2026-09-24 07:09:50 on dist-test-slave-2p7j"
server_key: "77e18e0f0f6163e1f01fda6822f665c0"
server_key_iv: "e27c4dec4155ac575674bb885e54b0dc"
server_key_version: "encryptionkey@0"
I20260924 07:09:50.607084 21822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "7fd533bd76f6421b94967a2d54012146"
format_stamp: "Formatted at 2026-09-24 07:09:50 on dist-test-slave-2p7j"
server_key: "77e18e0f0f6163e1f01fda6822f665c0"
server_key_iv: "e27c4dec4155ac575674bb885e54b0dc"
server_key_version: "encryptionkey@0"
I20260924 07:09:50.610388 21822 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 07:09:50.613137 21836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:50.614717 21822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 07:09:50.614849 21822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "7fd533bd76f6421b94967a2d54012146"
format_stamp: "Formatted at 2026-09-24 07:09:50 on dist-test-slave-2p7j"
server_key: "77e18e0f0f6163e1f01fda6822f665c0"
server_key_iv: "e27c4dec4155ac575674bb885e54b0dc"
server_key_version: "encryptionkey@0"
I20260924 07:09:50.614951 21822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:50.629078 21822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:50.629738 21822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:50.629899 21822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:50.630401 21822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:50.631254 21822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:50.631300 21822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:50.631350 21822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:50.631379 21822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:50.641579 21822 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:44841
I20260924 07:09:50.641690 21949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:44841 every 8 connection(s)
I20260924 07:09:50.642482 21822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:09:50.644474 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 21822
I20260924 07:09:50.644572 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:09:50.644819 3940 external_mini_cluster.cc:1428] Setting key 5dcba425254b49cbda35f04208dc4fea
I20260924 07:09:50.651537 21539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.644323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:54911 (local address 127.3.217.62:42751)
0924 07:09:50.644474 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:50.644478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:50.645507 (+ 1029us) server_negotiation.cc:407] Connection header received
0924 07:09:50.647074 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:50.647077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:50.647129 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:50.647195 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:50.648067 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.648610 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.649685 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.649899 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.649980 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.651064 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:50.651068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:50.651071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:50.651083 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:50.651102 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:50.651110 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:50.651167 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:50.651311 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 07:09:50.651368 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:09:50.652282 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.644691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42751 (local address 127.3.217.3:54911)
0924 07:09:50.645377 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:50.645426 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:50.646896 (+ 1470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:50.647352 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:50.647359 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:50.647435 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:50.647943 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:50.647955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.648720 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.648724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:50.649575 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:50.649585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.650023 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.650027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:50.650176 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.650811 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:09:50.650813 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:50.650854 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:09:50.650913 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:50.651178 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:50.651186 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:50.651192 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:09:50.651300 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 07:09:50.651602 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":121,"threads_started":1}
I20260924 07:09:50.653452 21950 heartbeater.cc:347] Connected to a master server at 127.3.217.62:42751
I20260924 07:09:50.653633 21950 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:50.654065 21950 heartbeater.cc:511] Master 127.3.217.62:42751 requested a full tablet report, sending...
I20260924 07:09:50.655004 21500 ts_manager.cc:194] Registered new tserver with Master: 7fd533bd76f6421b94967a2d54012146 (127.3.217.3:44841)
I20260924 07:09:50.655654 21500 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.217.3:54911
I20260924 07:09:50.658612 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:09:50.664384 21539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.660431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40796 (local address 127.3.217.62:42751)
0924 07:09:50.660683 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:50.660688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:50.661084 (+ 396us) server_negotiation.cc:407] Connection header received
0924 07:09:50.661158 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:50.661160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:50.661205 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:50.661268 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:50.661539 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.662039 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.662902 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.663123 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.663203 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.664006 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:50.664010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:50.664012 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:50.664024 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:50.664041 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:50.664048 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:50.664104 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:50.664190 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 07:09:50.664241 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260924 07:09:50.667111 21500 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:09:50.670035 21500 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:09:50.682621 21961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.677455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:44841 (local address 127.0.0.1:52738)
0924 07:09:50.678016 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:50.678050 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:50.678171 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:50.679231 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:50.679234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:50.679249 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:50.679329 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:50.679334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.680322 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.680325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:50.681290 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:50.681300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.682133 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.682137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:50.682235 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.682270 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 07:09:50.682375 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 07:09:50.682481 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":25,"spinlock_wait_cycles":2048,"thread_start_us":80,"threads_started":1}
I20260924 07:09:50.682626 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.678120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52738 (local address 127.3.217.3:44841)
0924 07:09:50.678588 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:50.678599 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:50.678621 (+ 22us) server_negotiation.cc:407] Connection header received
0924 07:09:50.678686 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:50.678691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:50.678910 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:50.679122 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:50.679432 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.680210 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.681409 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.682025 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.682120 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.682306 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:50.682400 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 07:09:50.682484 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":109,"threads_started":1}
I20260924 07:09:50.683089 21964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.677720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:44075 (local address 127.0.0.1:41250)
0924 07:09:50.678514 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:50.678526 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:50.678606 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:50.679174 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:50.679177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:50.679193 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:50.679281 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:50.679285 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.680264 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.680267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:50.681566 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:50.681574 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.682728 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.682732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:50.682822 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.682844 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:09:50.682905 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 07:09:50.682967 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":78,"thread_start_us":78,"threads_started":1}
I20260924 07:09:50.683576 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.678118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41250 (local address 127.3.217.1:44075)
0924 07:09:50.678419 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:50.678433 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:50.678687 (+ 254us) server_negotiation.cc:407] Connection header received
0924 07:09:50.678752 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:50.678756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:50.678907 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:50.679045 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:50.679392 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.680155 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.681678 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.682428 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.682518 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.682672 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:50.683298 (+ 626us) server_negotiation.cc:299] Negotiation successful
0924 07:09:50.683400 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":109,"threads_started":1}
I20260924 07:09:50.684190 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.677414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:42175 (local address 127.0.0.1:48830)
0924 07:09:50.678000 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:50.678046 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:50.678171 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:50.679100 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:50.679106 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:50.679129 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:50.679278 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:50.679285 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.681499 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.681503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:50.682574 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:50.682584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.683810 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.683814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:50.683907 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.683930 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:09:50.684006 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 07:09:50.684069 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":3,"spinlock_wait_cycles":6912,"thread_start_us":62,"threads_started":1}
I20260924 07:09:50.684281 21963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.677545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48830 (local address 127.3.217.2:42175)
0924 07:09:50.678613 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:50.678623 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:50.678645 (+ 22us) server_negotiation.cc:407] Connection header received
0924 07:09:50.678717 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:50.678722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:50.678907 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:50.679069 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:50.680280 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.681385 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.682842 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.683577 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.683673 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.683841 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:50.684045 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 07:09:50.684130 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":977,"thread_start_us":100,"threads_started":1}
I20260924 07:09:50.685716 21622 tablet_service.cc:1467] Processing CreateTablet for tablet e0a70517429f4bb5a59cfd08b7c525ea (DEFAULT_TABLE table=test-table [id=a2d155bab037476f942439af14e9672a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:50.685600 21884 tablet_service.cc:1467] Processing CreateTablet for tablet e0a70517429f4bb5a59cfd08b7c525ea (DEFAULT_TABLE table=test-table [id=a2d155bab037476f942439af14e9672a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:50.686619 21884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a70517429f4bb5a59cfd08b7c525ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:50.686619 21622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a70517429f4bb5a59cfd08b7c525ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:50.686620 21753 tablet_service.cc:1467] Processing CreateTablet for tablet e0a70517429f4bb5a59cfd08b7c525ea (DEFAULT_TABLE table=test-table [id=a2d155bab037476f942439af14e9672a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:50.687467 21753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a70517429f4bb5a59cfd08b7c525ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:50.692698 21967 tablet_bootstrap.cc:492] T e0a70517429f4bb5a59cfd08b7c525ea P eaab69c78ead4929be5906b473bddb1a: Bootstrap starting.
I20260924 07:09:50.694254 21969 tablet_bootstrap.cc:492] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184: Bootstrap starting.
I20260924 07:09:50.694785 21967 tablet_bootstrap.cc:654] T e0a70517429f4bb5a59cfd08b7c525ea P eaab69c78ead4929be5906b473bddb1a: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:50.695556 21967 log.cc:824] T e0a70517429f4bb5a59cfd08b7c525ea P eaab69c78ead4929be5906b473bddb1a: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:50.695811 21968 tablet_bootstrap.cc:492] T e0a70517429f4bb5a59cfd08b7c525ea P 7fd533bd76f6421b94967a2d54012146: Bootstrap starting.
I20260924 07:09:50.695964 21969 tablet_bootstrap.cc:654] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:50.696645 21969 log.cc:824] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:50.697451 21967 tablet_bootstrap.cc:492] T e0a70517429f4bb5a59cfd08b7c525ea P eaab69c78ead4929be5906b473bddb1a: No bootstrap required, opened a new log
I20260924 07:09:50.697485 21968 tablet_bootstrap.cc:654] T e0a70517429f4bb5a59cfd08b7c525ea P 7fd533bd76f6421b94967a2d54012146: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:50.697631 21967 ts_tablet_manager.cc:1397] T e0a70517429f4bb5a59cfd08b7c525ea P eaab69c78ead4929be5906b473bddb1a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 07:09:50.698069 21968 log.cc:824] T e0a70517429f4bb5a59cfd08b7c525ea P 7fd533bd76f6421b94967a2d54012146: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:50.698374 21969 tablet_bootstrap.cc:492] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184: No bootstrap required, opened a new log
I20260924 07:09:50.698534 21969 ts_tablet_manager.cc:1397] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 07:09:50.699404 21968 tablet_bootstrap.cc:492] T e0a70517429f4bb5a59cfd08b7c525ea P 7fd533bd76f6421b94967a2d54012146: No bootstrap required, opened a new log
I20260924 07:09:50.699571 21968 ts_tablet_manager.cc:1397] T e0a70517429f4bb5a59cfd08b7c525ea P 7fd533bd76f6421b94967a2d54012146: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260924 07:09:50.701767 21967 raft_consensus.cc:348] T e0a70517429f4bb5a59cfd08b7c525ea P eaab69c78ead4929be5906b473bddb1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87442f63bb44237a5007e749930c184" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 42175 } } peers { permanent_uuid: "7fd533bd76f6421b94967a2d54012146" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 44841 } } peers { permanent_uuid: "eaab69c78ead4929be5906b473bddb1a" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 44075 } }
I20260924 07:09:50.702016 21967 raft_consensus.cc:374] T e0a70517429f4bb5a59cfd08b7c525ea P eaab69c78ead4929be5906b473bddb1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:50.702072 21967 raft_consensus.cc:729] T e0a70517429f4bb5a59cfd08b7c525ea P eaab69c78ead4929be5906b473bddb1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaab69c78ead4929be5906b473bddb1a, State: Initialized, Role: FOLLOWER
I20260924 07:09:50.702311 21969 raft_consensus.cc:348] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87442f63bb44237a5007e749930c184" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 42175 } } peers { permanent_uuid: "7fd533bd76f6421b94967a2d54012146" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 44841 } } peers { permanent_uuid: "eaab69c78ead4929be5906b473bddb1a" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 44075 } }
I20260924 07:09:50.702525 21969 raft_consensus.cc:374] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:50.702489 21967 consensus_queue.cc:260] T e0a70517429f4bb5a59cfd08b7c525ea P eaab69c78ead4929be5906b473bddb1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87442f63bb44237a5007e749930c184" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 42175 } } peers { permanent_uuid: "7fd533bd76f6421b94967a2d54012146" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 44841 } } peers { permanent_uuid: "eaab69c78ead4929be5906b473bddb1a" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 44075 } }
I20260924 07:09:50.702571 21969 raft_consensus.cc:729] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87442f63bb44237a5007e749930c184, State: Initialized, Role: FOLLOWER
I20260924 07:09:50.702859 21968 raft_consensus.cc:348] T e0a70517429f4bb5a59cfd08b7c525ea P 7fd533bd76f6421b94967a2d54012146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87442f63bb44237a5007e749930c184" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 42175 } } peers { permanent_uuid: "7fd533bd76f6421b94967a2d54012146" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 44841 } } peers { permanent_uuid: "eaab69c78ead4929be5906b473bddb1a" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 44075 } }
I20260924 07:09:50.702970 21969 consensus_queue.cc:260] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87442f63bb44237a5007e749930c184" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 42175 } } peers { permanent_uuid: "7fd533bd76f6421b94967a2d54012146" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 44841 } } peers { permanent_uuid: "eaab69c78ead4929be5906b473bddb1a" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 44075 } }
I20260924 07:09:50.703081 21968 raft_consensus.cc:374] T e0a70517429f4bb5a59cfd08b7c525ea P 7fd533bd76f6421b94967a2d54012146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:50.703131 21968 raft_consensus.cc:729] T e0a70517429f4bb5a59cfd08b7c525ea P 7fd533bd76f6421b94967a2d54012146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd533bd76f6421b94967a2d54012146, State: Initialized, Role: FOLLOWER
I20260924 07:09:50.703296 21688 heartbeater.cc:503] Master 127.3.217.62:42751 was elected leader, sending a full tablet report...
I20260924 07:09:50.703402 21967 ts_tablet_manager.cc:1428] T e0a70517429f4bb5a59cfd08b7c525ea P eaab69c78ead4929be5906b473bddb1a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 07:09:50.703527 21968 consensus_queue.cc:260] T e0a70517429f4bb5a59cfd08b7c525ea P 7fd533bd76f6421b94967a2d54012146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87442f63bb44237a5007e749930c184" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 42175 } } peers { permanent_uuid: "7fd533bd76f6421b94967a2d54012146" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 44841 } } peers { permanent_uuid: "eaab69c78ead4929be5906b473bddb1a" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 44075 } }
I20260924 07:09:50.704087 21950 heartbeater.cc:503] Master 127.3.217.62:42751 was elected leader, sending a full tablet report...
I20260924 07:09:50.704334 21968 ts_tablet_manager.cc:1428] T e0a70517429f4bb5a59cfd08b7c525ea P 7fd533bd76f6421b94967a2d54012146: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:09:50.704704 21969 ts_tablet_manager.cc:1428] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 07:09:50.704967 21819 heartbeater.cc:503] Master 127.3.217.62:42751 was elected leader, sending a full tablet report...
W20260924 07:09:50.714707 21820 tablet.cc:2369] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:09:50.795845 21689 tablet.cc:2369] T e0a70517429f4bb5a59cfd08b7c525ea P eaab69c78ead4929be5906b473bddb1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:09:50.893633 21951 tablet.cc:2369] T e0a70517429f4bb5a59cfd08b7c525ea P 7fd533bd76f6421b94967a2d54012146: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:09:50.902858 21975 raft_consensus.cc:482] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:09:50.903086 21975 raft_consensus.cc:504] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87442f63bb44237a5007e749930c184" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 42175 } } peers { permanent_uuid: "7fd533bd76f6421b94967a2d54012146" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 44841 } } peers { permanent_uuid: "eaab69c78ead4929be5906b473bddb1a" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 44075 } }
I20260924 07:09:50.904316 21975 leader_election.cc:288] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fd533bd76f6421b94967a2d54012146 (127.3.217.3:44841), eaab69c78ead4929be5906b473bddb1a (127.3.217.1:44075)
I20260924 07:09:50.909004 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.904861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:36531 (local address 127.3.217.1:44075)
0924 07:09:50.905015 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:50.905020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:50.905041 (+ 21us) server_negotiation.cc:407] Connection header received
0924 07:09:50.905165 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:50.905169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:50.905228 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:50.905353 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:50.905776 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.906389 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.907849 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.908331 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.908411 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.908461 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:50.908795 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 07:09:50.908868 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 07:09:50.909014 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.904737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:44075 (local address 127.3.217.2:36531)
0924 07:09:50.904879 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:50.904899 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:50.905037 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:50.905407 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:50.905410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:50.905433 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:50.905629 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:50.905640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.906510 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.906514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:50.907715 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:50.907725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.908470 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.908474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:50.908586 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.908667 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 07:09:50.908763 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 07:09:50.908861 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"spinlock_wait_cycles":1280}
I20260924 07:09:50.909288 21976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.904737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:44841 (local address 127.3.217.2:49479)
0924 07:09:50.905360 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:50.905378 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:50.905512 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:50.905809 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:50.905812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:50.905834 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:50.905978 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:50.905986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.906991 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.906994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:50.907984 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:50.907994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.908814 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.908817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:50.908919 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.908948 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 07:09:50.909031 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:09:50.909105 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":198,"thread_start_us":156,"threads_started":1}
I20260924 07:09:50.909296 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.904861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:49479 (local address 127.3.217.3:44841)
0924 07:09:50.905015 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:50.905020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:50.905495 (+ 475us) server_negotiation.cc:407] Connection header received
0924 07:09:50.905631 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:50.905635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:50.905695 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:50.905777 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:50.906109 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.906830 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.908141 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.908684 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.908784 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.908850 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:50.909060 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 07:09:50.909162 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 07:09:50.909910 21642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a70517429f4bb5a59cfd08b7c525ea" candidate_uuid: "c87442f63bb44237a5007e749930c184" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaab69c78ead4929be5906b473bddb1a" is_pre_election: true
I20260924 07:09:50.910018 21904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a70517429f4bb5a59cfd08b7c525ea" candidate_uuid: "c87442f63bb44237a5007e749930c184" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd533bd76f6421b94967a2d54012146" is_pre_election: true
I20260924 07:09:50.910277 21642 raft_consensus.cc:2454] T e0a70517429f4bb5a59cfd08b7c525ea P eaab69c78ead4929be5906b473bddb1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87442f63bb44237a5007e749930c184 in term 0.
I20260924 07:09:50.910295 21904 raft_consensus.cc:2454] T e0a70517429f4bb5a59cfd08b7c525ea P 7fd533bd76f6421b94967a2d54012146 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87442f63bb44237a5007e749930c184 in term 0.
I20260924 07:09:50.910838 21707 leader_election.cc:302] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fd533bd76f6421b94967a2d54012146, c87442f63bb44237a5007e749930c184; no voters:
I20260924 07:09:50.911159 21975 raft_consensus.cc:2781] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:09:50.911264 21975 raft_consensus.cc:482] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:09:50.911302 21975 raft_consensus.cc:3037] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:50.912460 21975 raft_consensus.cc:504] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87442f63bb44237a5007e749930c184" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 42175 } } peers { permanent_uuid: "7fd533bd76f6421b94967a2d54012146" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 44841 } } peers { permanent_uuid: "eaab69c78ead4929be5906b473bddb1a" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 44075 } }
I20260924 07:09:50.912843 21975 leader_election.cc:288] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184 [CANDIDATE]: Term 1 election: Requested vote from peers 7fd533bd76f6421b94967a2d54012146 (127.3.217.3:44841), eaab69c78ead4929be5906b473bddb1a (127.3.217.1:44075)
I20260924 07:09:50.913247 21642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a70517429f4bb5a59cfd08b7c525ea" candidate_uuid: "c87442f63bb44237a5007e749930c184" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaab69c78ead4929be5906b473bddb1a"
I20260924 07:09:50.913239 21904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a70517429f4bb5a59cfd08b7c525ea" candidate_uuid: "c87442f63bb44237a5007e749930c184" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd533bd76f6421b94967a2d54012146"
I20260924 07:09:50.913386 21642 raft_consensus.cc:3037] T e0a70517429f4bb5a59cfd08b7c525ea P eaab69c78ead4929be5906b473bddb1a [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:50.913401 21904 raft_consensus.cc:3037] T e0a70517429f4bb5a59cfd08b7c525ea P 7fd533bd76f6421b94967a2d54012146 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:50.914307 21642 raft_consensus.cc:2454] T e0a70517429f4bb5a59cfd08b7c525ea P eaab69c78ead4929be5906b473bddb1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87442f63bb44237a5007e749930c184 in term 1.
I20260924 07:09:50.914626 21709 leader_election.cc:302] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c87442f63bb44237a5007e749930c184, eaab69c78ead4929be5906b473bddb1a; no voters:
I20260924 07:09:50.914798 21975 raft_consensus.cc:2781] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:50.914803 21904 raft_consensus.cc:2454] T e0a70517429f4bb5a59cfd08b7c525ea P 7fd533bd76f6421b94967a2d54012146 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87442f63bb44237a5007e749930c184 in term 1.
I20260924 07:09:50.915014 21975 raft_consensus.cc:686] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184 [term 1 LEADER]: Becoming Leader. State: Replica: c87442f63bb44237a5007e749930c184, State: Running, Role: LEADER
I20260924 07:09:50.915333 21975 consensus_queue.cc:237] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87442f63bb44237a5007e749930c184" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 42175 } } peers { permanent_uuid: "7fd533bd76f6421b94967a2d54012146" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 44841 } } peers { permanent_uuid: "eaab69c78ead4929be5906b473bddb1a" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 44075 } }
I20260924 07:09:50.918927 21499 catalog_manager.cc:5495] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184 reported cstate change: term changed from 0 to 1, leader changed from <none> to c87442f63bb44237a5007e749930c184 (127.3.217.2). New cstate: current_term: 1 leader_uuid: "c87442f63bb44237a5007e749930c184" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87442f63bb44237a5007e749930c184" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 42175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fd533bd76f6421b94967a2d54012146" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 44841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaab69c78ead4929be5906b473bddb1a" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 44075 } health_report { overall_health: UNKNOWN } } }
I20260924 07:09:50.979249 21963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.976015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48842 (local address 127.3.217.2:42175)
0924 07:09:50.976181 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:50.976185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:50.976199 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:09:50.976269 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:50.976271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:50.976343 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:50.976431 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:09:50.976653 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.977140 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.977946 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.978163 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.978245 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.978515 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:09:50.978892 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:09:50.978990 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:50.979059 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 07:09:50.979109 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 07:09:50.986078 21642 raft_consensus.cc:1261] T e0a70517429f4bb5a59cfd08b7c525ea P eaab69c78ead4929be5906b473bddb1a [term 1 FOLLOWER]: Refusing update from remote peer c87442f63bb44237a5007e749930c184: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:50.986153 21904 raft_consensus.cc:1261] T e0a70517429f4bb5a59cfd08b7c525ea P 7fd533bd76f6421b94967a2d54012146 [term 1 FOLLOWER]: Refusing update from remote peer c87442f63bb44237a5007e749930c184: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:50.986905 21977 consensus_queue.cc:1035] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd533bd76f6421b94967a2d54012146" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 44841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:50.987124 21975 consensus_queue.cc:1035] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaab69c78ead4929be5906b473bddb1a" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 44075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:50.997138 21982 mvcc.cc:205] Tried to move back new op lower bound from 7332797607871127552 to 7332797607591538688. Current Snapshot: MvccSnapshot[applied={T|T < 7332797607871127552}]
I20260924 07:09:51.006389 21499 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:51.006620 21499 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:51.009678 21499 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c45e65f5b1345e8838e40008b29cc46: Sending DeleteTablet for 3 replicas of tablet e0a70517429f4bb5a59cfd08b7c525ea
I20260924 07:09:51.010696 21753 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a70517429f4bb5a59cfd08b7c525ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:51 UTC) from {username='slave'} at 127.0.0.1:48830
I20260924 07:09:51.010696 21884 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a70517429f4bb5a59cfd08b7c525ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:51 UTC) from {username='slave'} at 127.0.0.1:52738
I20260924 07:09:51.010738 21622 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a70517429f4bb5a59cfd08b7c525ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:51 UTC) from {username='slave'} at 127.0.0.1:41250
I20260924 07:09:51.011366 21991 tablet_replica.cc:330] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184: stopping tablet replica
I20260924 07:09:51.011377 21992 tablet_replica.cc:330] T e0a70517429f4bb5a59cfd08b7c525ea P eaab69c78ead4929be5906b473bddb1a: stopping tablet replica
I20260924 07:09:51.011693 21990 tablet_replica.cc:330] T e0a70517429f4bb5a59cfd08b7c525ea P 7fd533bd76f6421b94967a2d54012146: stopping tablet replica
I20260924 07:09:51.011845 21992 raft_consensus.cc:2229] T e0a70517429f4bb5a59cfd08b7c525ea P eaab69c78ead4929be5906b473bddb1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:51.011946 21990 raft_consensus.cc:2229] T e0a70517429f4bb5a59cfd08b7c525ea P 7fd533bd76f6421b94967a2d54012146 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:51.012277 21990 raft_consensus.cc:2258] T e0a70517429f4bb5a59cfd08b7c525ea P 7fd533bd76f6421b94967a2d54012146 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:51.012279 21992 raft_consensus.cc:2258] T e0a70517429f4bb5a59cfd08b7c525ea P eaab69c78ead4929be5906b473bddb1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:51.012509 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 21560
I20260924 07:09:51.012739 21991 raft_consensus.cc:2229] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:09:51.013206 21991 raft_consensus.cc:2258] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:51.014389 21990 ts_tablet_manager.cc:1905] T e0a70517429f4bb5a59cfd08b7c525ea P 7fd533bd76f6421b94967a2d54012146: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:51.017169 21991 ts_tablet_manager.cc:1905] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:51.017937 21990 ts_tablet_manager.cc:1918] T e0a70517429f4bb5a59cfd08b7c525ea P 7fd533bd76f6421b94967a2d54012146: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:09:51.018061 21990 log.cc:1196] T e0a70517429f4bb5a59cfd08b7c525ea P 7fd533bd76f6421b94967a2d54012146: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/e0a70517429f4bb5a59cfd08b7c525ea
I20260924 07:09:51.018375 21990 ts_tablet_manager.cc:1939] T e0a70517429f4bb5a59cfd08b7c525ea P 7fd533bd76f6421b94967a2d54012146: Deleting consensus metadata
I20260924 07:09:51.020006 21485 catalog_manager.cc:4841] TS 7fd533bd76f6421b94967a2d54012146 (127.3.217.3:44841): tablet e0a70517429f4bb5a59cfd08b7c525ea (table test-table [id=a2d155bab037476f942439af14e9672a]) successfully deleted
I20260924 07:09:51.020803 21991 ts_tablet_manager.cc:1918] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:09:51.020907 21991 log.cc:1196] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/e0a70517429f4bb5a59cfd08b7c525ea
I20260924 07:09:51.021250 21991 ts_tablet_manager.cc:1939] T e0a70517429f4bb5a59cfd08b7c525ea P c87442f63bb44237a5007e749930c184: Deleting consensus metadata
W20260924 07:09:51.021584 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.217.1:44075 (error 108)
I20260924 07:09:51.022353 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 21691
I20260924 07:09:51.023962 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:51.023638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:44075 (local address 127.0.0.1:41262)
0924 07:09:51.023817 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:51.023880 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.1:44075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260924 07:09:51.024112 21487 catalog_manager.cc:4584] TS eaab69c78ead4929be5906b473bddb1a (127.3.217.1:44075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0a70517429f4bb5a59cfd08b7c525ea: Network error: Client connection negotiation failed: client connection to 127.3.217.1:44075: connect: Connection refused (error 111)
I20260924 07:09:51.029072 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 21822
I20260924 07:09:51.029631 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:51.029394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:42175 (local address 127.0.0.1:48852)
0924 07:09:51.029509 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:51.029568 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.2:42175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 07:09:51.036078 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 21469
2026-09-24T07:09:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1211 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:09:51 dist-test-slave-2p7j krb5kdc[21996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:09:51 dist-test-slave-2p7j krb5kdc[21996](info): set up 2 sockets
Sep 24 07:09:51 dist-test-slave-2p7j krb5kdc[21996](info): commencing operation
krb5kdc: starting...
W20260924 07:09:52.000278 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:09:52 dist-test-slave-2p7j krb5kdc[21996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:09:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:09:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:09:52.122226 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:41751
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:41751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:52.232820 22009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:52.233060 22009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:52.233114 22009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:52.238334 22009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:09:52.238451 22009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:52.238478 22009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:52.238502 22009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:09:52.238526 22009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:09:52.245488 22009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:41751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:41751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:52.246541 22009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:52.247632 22009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:52.256798 22015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:52.256817 22014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:52.256852 22017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:52.257292 22009 server_base.cc:1034] running on GCE node
I20260924 07:09:52.257815 22009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:52.258847 22009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:52.260030 22009 hybrid_clock.cc:648] HybridClock initialized: now 1790233792260006 us; error 33 us; skew 500 ppm
Sep 24 07:09:52 dist-test-slave-2p7j krb5kdc[21996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233792, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:52.262753 22009 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:09:52.263595 22009 webserver.cc:460] Webserver started at http://127.3.217.62:36177/ using document root <none> and password file <none>
I20260924 07:09:52.264071 22009 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:52.264127 22009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:52.264283 22009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:52.266134 22009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "a1cf1723ecfd4ec0a5675a7a9b8faf72"
format_stamp: "Formatted at 2026-09-24 07:09:52 on dist-test-slave-2p7j"
server_key: "df4599069a521a26c5e210a8d5b82a45"
server_key_iv: "d2c01c039437c24e68873d79b1d0cd36"
server_key_version: "encryptionkey@0"
I20260924 07:09:52.266620 22009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "a1cf1723ecfd4ec0a5675a7a9b8faf72"
format_stamp: "Formatted at 2026-09-24 07:09:52 on dist-test-slave-2p7j"
server_key: "df4599069a521a26c5e210a8d5b82a45"
server_key_iv: "d2c01c039437c24e68873d79b1d0cd36"
server_key_version: "encryptionkey@0"
I20260924 07:09:52.270018 22009 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 07:09:52.272753 22024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:52.273928 22009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:09:52.274025 22009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "a1cf1723ecfd4ec0a5675a7a9b8faf72"
format_stamp: "Formatted at 2026-09-24 07:09:52 on dist-test-slave-2p7j"
server_key: "df4599069a521a26c5e210a8d5b82a45"
server_key_iv: "d2c01c039437c24e68873d79b1d0cd36"
server_key_version: "encryptionkey@0"
I20260924 07:09:52.274106 22009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:52.296312 22009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:52.299223 22009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:52.299368 22009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:52.305822 22009 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:41751
I20260924 07:09:52.305851 22076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:41751 every 8 connection(s)
I20260924 07:09:52.307279 22009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:09:52.307605 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 22009
I20260924 07:09:52.307713 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:09:52.308024 3940 external_mini_cluster.cc:1428] Setting key f56fb32cb078300cefc83a82ff92006f
I20260924 07:09:52.310278 22077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:52.316587 22077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1cf1723ecfd4ec0a5675a7a9b8faf72: Bootstrap starting.
Sep 24 07:09:52 dist-test-slave-2p7j krb5kdc[21996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:09:52.318557 22077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1cf1723ecfd4ec0a5675a7a9b8faf72: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:52.319229 22077 log.cc:824] T 00000000000000000000000000000000 P a1cf1723ecfd4ec0a5675a7a9b8faf72: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:52.320904 22077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1cf1723ecfd4ec0a5675a7a9b8faf72: No bootstrap required, opened a new log
I20260924 07:09:52.321249 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:52.309261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36308 (local address 127.3.217.62:41751)
0924 07:09:52.309645 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:52.309659 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:52.309691 (+ 32us) server_negotiation.cc:407] Connection header received
0924 07:09:52.310296 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:52.310310 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:52.310622 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:52.310871 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:52.312082 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:52.313046 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:52.314162 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:52.314609 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:52.314723 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:52.317293 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:52.317298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:52.317308 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:52.317333 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:52.318811 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:52.319181 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:52.319186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:52.319190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:52.319302 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:52.319553 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:52.319556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:52.319558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:52.319918 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:52.320085 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:52.320361 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 07:09:52.320598 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":117,"threads_started":1}
I20260924 07:09:52.323890 22077 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1cf1723ecfd4ec0a5675a7a9b8faf72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cf1723ecfd4ec0a5675a7a9b8faf72" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 41751 } }
I20260924 07:09:52.324115 22077 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1cf1723ecfd4ec0a5675a7a9b8faf72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:52.324168 22077 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1cf1723ecfd4ec0a5675a7a9b8faf72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1cf1723ecfd4ec0a5675a7a9b8faf72, State: Initialized, Role: FOLLOWER
I20260924 07:09:52.324601 22077 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1cf1723ecfd4ec0a5675a7a9b8faf72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cf1723ecfd4ec0a5675a7a9b8faf72" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 41751 } }
I20260924 07:09:52.324733 22077 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1cf1723ecfd4ec0a5675a7a9b8faf72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:09:52.324796 22077 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1cf1723ecfd4ec0a5675a7a9b8faf72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:09:52.324890 22077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1cf1723ecfd4ec0a5675a7a9b8faf72 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:52.325795 22077 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1cf1723ecfd4ec0a5675a7a9b8faf72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cf1723ecfd4ec0a5675a7a9b8faf72" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 41751 } }
I20260924 07:09:52.326122 22077 leader_election.cc:302] T 00000000000000000000000000000000 P a1cf1723ecfd4ec0a5675a7a9b8faf72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1cf1723ecfd4ec0a5675a7a9b8faf72; no voters:
I20260924 07:09:52.326517 22082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1cf1723ecfd4ec0a5675a7a9b8faf72 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:52.326436 22077 leader_election.cc:288] T 00000000000000000000000000000000 P a1cf1723ecfd4ec0a5675a7a9b8faf72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:09:52.326789 22082 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1cf1723ecfd4ec0a5675a7a9b8faf72 [term 1 LEADER]: Becoming Leader. State: Replica: a1cf1723ecfd4ec0a5675a7a9b8faf72, State: Running, Role: LEADER
I20260924 07:09:52.327054 22082 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1cf1723ecfd4ec0a5675a7a9b8faf72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cf1723ecfd4ec0a5675a7a9b8faf72" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 41751 } }
I20260924 07:09:52.327898 22077 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1cf1723ecfd4ec0a5675a7a9b8faf72 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:09:52.328742 22083 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1cf1723ecfd4ec0a5675a7a9b8faf72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1cf1723ecfd4ec0a5675a7a9b8faf72. Latest consensus state: current_term: 1 leader_uuid: "a1cf1723ecfd4ec0a5675a7a9b8faf72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cf1723ecfd4ec0a5675a7a9b8faf72" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 41751 } } }
I20260924 07:09:52.328856 22083 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1cf1723ecfd4ec0a5675a7a9b8faf72 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:52.329962 22089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:09:52.328830 22084 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1cf1723ecfd4ec0a5675a7a9b8faf72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1cf1723ecfd4ec0a5675a7a9b8faf72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cf1723ecfd4ec0a5675a7a9b8faf72" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 41751 } } }
I20260924 07:09:52.330096 22084 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1cf1723ecfd4ec0a5675a7a9b8faf72 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:52.332949 22089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:09:52.338233 22089 catalog_manager.cc:1347] Generated new cluster ID: c6bd111ac5d9424a9732d76e0bf47f55
I20260924 07:09:52.338316 22089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:09:52.346342 22089 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:09:52.347158 22089 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:09:52.353152 22089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1cf1723ecfd4ec0a5675a7a9b8faf72: Generated new TSK 0
I20260924 07:09:52.353771 22089 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
I20260924 07:09:52.387264 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:41751
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:52.495846 22103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:52.496064 22103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:52.496119 22103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:52.500171 22103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:52.500229 22103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:52.500310 22103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:09:52.505376 22103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:41751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:52.506244 22103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:52.507129 22103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:52.515950 22111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:52.515936 22108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:52.515939 22109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:52.516307 22103 server_base.cc:1034] running on GCE node
I20260924 07:09:52.516623 22103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:52.517129 22103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:52.518311 22103 hybrid_clock.cc:648] HybridClock initialized: now 1790233792518292 us; error 35 us; skew 500 ppm
Sep 24 07:09:52 dist-test-slave-2p7j krb5kdc[21996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233792, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:52.521453 22103 init.cc:377] Logged in from keytab as kudu/127.3.217.1@KRBTEST.COM (short username kudu)
I20260924 07:09:52.522280 22103 webserver.cc:460] Webserver started at http://127.3.217.1:36483/ using document root <none> and password file <none>
I20260924 07:09:52.522814 22103 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:52.522886 22103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:52.523078 22103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:52.525817 22103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "483c017124c3480d9cdd1d570706f0f0"
format_stamp: "Formatted at 2026-09-24 07:09:52 on dist-test-slave-2p7j"
server_key: "40764d3e504a0d88c07cb63b8fb32e13"
server_key_iv: "4ba71fd1b46058a1c4f7438da7fc3f88"
server_key_version: "encryptionkey@0"
I20260924 07:09:52.526746 22103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "483c017124c3480d9cdd1d570706f0f0"
format_stamp: "Formatted at 2026-09-24 07:09:52 on dist-test-slave-2p7j"
server_key: "40764d3e504a0d88c07cb63b8fb32e13"
server_key_iv: "4ba71fd1b46058a1c4f7438da7fc3f88"
server_key_version: "encryptionkey@0"
I20260924 07:09:52.530694 22103 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:09:52.533047 22118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:52.534111 22103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:09:52.534212 22103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "483c017124c3480d9cdd1d570706f0f0"
format_stamp: "Formatted at 2026-09-24 07:09:52 on dist-test-slave-2p7j"
server_key: "40764d3e504a0d88c07cb63b8fb32e13"
server_key_iv: "4ba71fd1b46058a1c4f7438da7fc3f88"
server_key_version: "encryptionkey@0"
I20260924 07:09:52.534307 22103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:52.557370 22103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:52.560288 22103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:52.560454 22103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:52.560983 22103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:52.561741 22103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:52.561787 22103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:52.561839 22103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:52.561865 22103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:52.570185 22103 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:42183
I20260924 07:09:52.570226 22231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:42183 every 8 connection(s)
I20260924 07:09:52.570981 22103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:09:52.573361 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 22103
I20260924 07:09:52.573490 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:09:52.573777 3940 external_mini_cluster.cc:1428] Setting key 6a5c67147a6027a2ea569c11a5990439
Sep 24 07:09:52 dist-test-slave-2p7j krb5kdc[21996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233792, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.217.2@KRBTEST.COM; defaulting to no policy
I20260924 07:09:52.584367 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:52.573140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:59439 (local address 127.3.217.62:41751)
0924 07:09:52.573297 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:52.573302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:52.573954 (+ 652us) server_negotiation.cc:407] Connection header received
0924 07:09:52.575370 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:52.575373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:52.575425 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:52.575493 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:52.576598 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:52.577163 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:52.578331 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:52.578617 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:52.578709 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:52.580981 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:52.580984 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:52.580987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:52.581012 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:52.582218 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:52.582683 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:52.582686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:52.582687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:52.582758 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:52.583049 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:52.583053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:52.583054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:52.583253 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:52.583345 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:52.583987 (+ 642us) server_negotiation.cc:299] Negotiation successful
0924 07:09:52.584093 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.2@KRBTEST.COM" created.
I20260924 07:09:52.584949 22234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:52.573344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:41751 (local address 127.3.217.1:59439)
0924 07:09:52.573804 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:52.573852 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:52.575150 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:52.575669 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:52.575677 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:52.575998 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:52.576461 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:52.576474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:52.577281 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:52.577284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:52.578206 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:52.578216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:52.578756 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:52.578761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:52.578909 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:52.579405 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:52.579420 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:52.580833 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:52.582325 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:52.582330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:52.582343 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:52.582586 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:52.582837 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:52.582840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:52.582842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:52.582966 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:52.583491 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:52.583496 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:52.583762 (+ 266us) client_negotiation.cc:769] Sending connection context
0924 07:09:52.583923 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 07:09:52.584154 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":119,"threads_started":1}
I20260924 07:09:52.586092 22232 heartbeater.cc:347] Connected to a master server at 127.3.217.62:41751
I20260924 07:09:52.586314 22232 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:52.586740 22232 heartbeater.cc:511] Master 127.3.217.62:41751 requested a full tablet report, sending...
I20260924 07:09:52.588105 22041 ts_manager.cc:194] Registered new tserver with Master: 483c017124c3480d9cdd1d570706f0f0 (127.3.217.1:42183)
I20260924 07:09:52.589134 22041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.1@KRBTEST.COM'} at 127.3.217.1:59439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
I20260924 07:09:52.599932 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:41751
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:52.703724 22237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:52.703946 22237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:52.703999 22237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:52.707885 22237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:52.707931 22237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:52.707984 22237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:09:52.713073 22237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:41751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:52.713917 22237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:52.714850 22237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:52.723793 22242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:52.723806 22245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:52.723878 22237 server_base.cc:1034] running on GCE node
W20260924 07:09:52.723789 22243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:52.724481 22237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:52.725052 22237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:52.726229 22237 hybrid_clock.cc:648] HybridClock initialized: now 1790233792726209 us; error 26 us; skew 500 ppm
Sep 24 07:09:52 dist-test-slave-2p7j krb5kdc[21996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233792, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:52.728909 22237 init.cc:377] Logged in from keytab as kudu/127.3.217.2@KRBTEST.COM (short username kudu)
I20260924 07:09:52.729758 22237 webserver.cc:460] Webserver started at http://127.3.217.2:40473/ using document root <none> and password file <none>
I20260924 07:09:52.730293 22237 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:52.730361 22237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:52.730551 22237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:52.733186 22237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "9b00a054d5a7497385ad15fc9f67361d"
format_stamp: "Formatted at 2026-09-24 07:09:52 on dist-test-slave-2p7j"
server_key: "dd9e27bfbb2d2e3720bb27c529d3f567"
server_key_iv: "08397a591c87e2f83d443cdc2d13506f"
server_key_version: "encryptionkey@0"
I20260924 07:09:52.734000 22237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "9b00a054d5a7497385ad15fc9f67361d"
format_stamp: "Formatted at 2026-09-24 07:09:52 on dist-test-slave-2p7j"
server_key: "dd9e27bfbb2d2e3720bb27c529d3f567"
server_key_iv: "08397a591c87e2f83d443cdc2d13506f"
server_key_version: "encryptionkey@0"
I20260924 07:09:52.737470 22237 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:09:52.739732 22252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:52.740803 22237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:09:52.740906 22237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "9b00a054d5a7497385ad15fc9f67361d"
format_stamp: "Formatted at 2026-09-24 07:09:52 on dist-test-slave-2p7j"
server_key: "dd9e27bfbb2d2e3720bb27c529d3f567"
server_key_iv: "08397a591c87e2f83d443cdc2d13506f"
server_key_version: "encryptionkey@0"
I20260924 07:09:52.740998 22237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:52.755015 22237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:52.757485 22237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:52.757632 22237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:52.758112 22237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:52.758821 22237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:52.758865 22237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:52.758912 22237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:52.758941 22237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:52.766921 22237 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:39029
I20260924 07:09:52.767037 22365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:39029 every 8 connection(s)
I20260924 07:09:52.767752 22237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:09:52.775084 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 22237
I20260924 07:09:52.775202 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:09:52.775445 3940 external_mini_cluster.cc:1428] Setting key f7b40d959107041d0a910def03f9df4d
Sep 24 07:09:52 dist-test-slave-2p7j krb5kdc[21996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233792, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:09:52.780721 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:52.769394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:51277 (local address 127.3.217.62:41751)
0924 07:09:52.769528 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:52.769532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:52.770232 (+ 700us) server_negotiation.cc:407] Connection header received
0924 07:09:52.771408 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:52.771411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:52.771463 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:52.771549 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:52.772561 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:52.773074 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:52.774123 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:52.774335 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:52.774418 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:52.777250 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:52.777255 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:52.777258 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:52.777284 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:52.778702 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:52.779259 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:52.779263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:52.779265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:52.779340 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:52.779674 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:52.779678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:52.779680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:52.779849 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:52.779932 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:52.780469 (+ 537us) server_negotiation.cc:299] Negotiation successful
0924 07:09:52.780579 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:09:52.781550 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:52.769677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:41751 (local address 127.3.217.2:51277)
0924 07:09:52.770100 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:52.770139 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:52.771222 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:52.771698 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:52.771704 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:52.772015 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:52.772431 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:52.772443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:52.773186 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:52.773189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:52.774010 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:52.774020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:52.774462 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:52.774466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:52.774611 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:52.775098 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:52.775113 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:52.776912 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:52.778844 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:52.778850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:52.778865 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:52.779156 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:52.779419 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:52.779422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:52.779424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:52.779590 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:52.779944 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:52.779950 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:52.780203 (+ 253us) client_negotiation.cc:769] Sending connection context
0924 07:09:52.780374 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 07:09:52.780654 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":103,"threads_started":1}
I20260924 07:09:52.782631 22366 heartbeater.cc:347] Connected to a master server at 127.3.217.62:41751
I20260924 07:09:52.782866 22366 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:52.783370 22366 heartbeater.cc:511] Master 127.3.217.62:41751 requested a full tablet report, sending...
I20260924 07:09:52.784509 22041 ts_manager.cc:194] Registered new tserver with Master: 9b00a054d5a7497385ad15fc9f67361d (127.3.217.2:39029)
I20260924 07:09:52.785225 22041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.2@KRBTEST.COM'} at 127.3.217.2:51277
WARNING: no policy specified for kudu/127.3.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
I20260924 07:09:52.803244 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:41751
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:52.906342 22371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:52.906724 22371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:52.906792 22371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:52.911881 22371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:52.911942 22371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:52.912006 22371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:09:52.917718 22371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:41751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:52.918646 22371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:52.919602 22371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:52.930047 22376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:52.930094 22377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:52.930047 22379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:52.930565 22371 server_base.cc:1034] running on GCE node
I20260924 07:09:52.931085 22371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:52.931725 22371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:52.932914 22371 hybrid_clock.cc:648] HybridClock initialized: now 1790233792932892 us; error 37 us; skew 500 ppm
Sep 24 07:09:52 dist-test-slave-2p7j krb5kdc[21996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233792, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:52.935436 22371 init.cc:377] Logged in from keytab as kudu/127.3.217.3@KRBTEST.COM (short username kudu)
I20260924 07:09:52.936249 22371 webserver.cc:460] Webserver started at http://127.3.217.3:38455/ using document root <none> and password file <none>
I20260924 07:09:52.936738 22371 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:52.936798 22371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:52.936947 22371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:52.938875 22371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "865ecf37641e4ea6a5a076870cbfcfa9"
format_stamp: "Formatted at 2026-09-24 07:09:52 on dist-test-slave-2p7j"
server_key: "83d1e6306a967a3b51aa42ec7fd42312"
server_key_iv: "5f5d3b0ffd7e6f12d054a4a6e82e04fa"
server_key_version: "encryptionkey@0"
I20260924 07:09:52.939334 22371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "865ecf37641e4ea6a5a076870cbfcfa9"
format_stamp: "Formatted at 2026-09-24 07:09:52 on dist-test-slave-2p7j"
server_key: "83d1e6306a967a3b51aa42ec7fd42312"
server_key_iv: "5f5d3b0ffd7e6f12d054a4a6e82e04fa"
server_key_version: "encryptionkey@0"
I20260924 07:09:52.943674 22371 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:09:52.946563 22386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:52.948156 22371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:09:52.948287 22371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "865ecf37641e4ea6a5a076870cbfcfa9"
format_stamp: "Formatted at 2026-09-24 07:09:52 on dist-test-slave-2p7j"
server_key: "83d1e6306a967a3b51aa42ec7fd42312"
server_key_iv: "5f5d3b0ffd7e6f12d054a4a6e82e04fa"
server_key_version: "encryptionkey@0"
I20260924 07:09:52.948385 22371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:52.976882 22371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:52.980139 22371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:52.980296 22371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:52.980831 22371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:52.981578 22371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:52.981626 22371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:52.981662 22371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:52.981683 22371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:52.991436 22371 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:44005
I20260924 07:09:52.991459 22499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:44005 every 8 connection(s)
I20260924 07:09:52.992282 22371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:09:52.999264 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 22371
I20260924 07:09:52.999382 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:09:52.999681 3940 external_mini_cluster.cc:1428] Setting key a9fbcc1a40bc50117b8068c655fe0938
Sep 24 07:09:52 dist-test-slave-2p7j krb5kdc[21996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233792, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:09:53.005092 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:52.994564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:46265 (local address 127.3.217.62:41751)
0924 07:09:52.994711 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:52.994716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:52.995529 (+ 813us) server_negotiation.cc:407] Connection header received
0924 07:09:52.996333 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:52.996336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:52.996380 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:52.996449 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:52.997572 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:52.998093 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:52.999243 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:52.999521 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:52.999601 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.002051 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:53.002054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:53.002056 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:53.002080 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:53.003309 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:53.003811 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:53.003813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:53.003815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:53.003884 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:53.004178 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:53.004181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:53.004182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:53.004380 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:53.004476 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:53.004892 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 07:09:53.004982 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:09:53.005760 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:52.994925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:41751 (local address 127.3.217.3:46265)
0924 07:09:52.995356 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:52.995396 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:52.996160 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:52.996635 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:52.996642 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:52.996990 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:52.997435 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:52.997447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:52.998209 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:52.998212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:52.999109 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:52.999120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:52.999649 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:52.999669 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:52.999854 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.000453 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:53.000470 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:53.001903 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:53.003416 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:53.003420 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:53.003430 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:53.003724 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:53.003957 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:53.003959 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:53.003960 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:53.004078 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:53.004477 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:53.004481 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:53.004677 (+ 196us) client_negotiation.cc:769] Sending connection context
0924 07:09:53.004821 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 07:09:53.005038 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":102,"threads_started":1}
I20260924 07:09:53.006757 22500 heartbeater.cc:347] Connected to a master server at 127.3.217.62:41751
I20260924 07:09:53.006959 22500 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:53.007372 22500 heartbeater.cc:511] Master 127.3.217.62:41751 requested a full tablet report, sending...
I20260924 07:09:53.008549 22041 ts_manager.cc:194] Registered new tserver with Master: 865ecf37641e4ea6a5a076870cbfcfa9 (127.3.217.3:44005)
I20260924 07:09:53.009346 22041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.3@KRBTEST.COM'} at 127.3.217.3:46265
I20260924 07:09:53.013466 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:09:53.024425 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.015766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36310 (local address 127.3.217.62:41751)
0924 07:09:53.015875 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:53.015881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:53.016000 (+ 119us) server_negotiation.cc:407] Connection header received
0924 07:09:53.016111 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:53.016115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:53.016155 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:53.016218 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:53.016842 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.017560 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.018449 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.018665 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.018738 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.020269 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:53.020274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:53.020276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:53.020314 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:53.022147 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:53.022861 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:53.022867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:53.022869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:53.023010 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:53.023498 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:53.023534 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:53.023536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:53.023776 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:53.023911 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:53.024202 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 07:09:53.024293 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:09:53.027275 22041 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:09:53.030726 22041 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:09:53.044992 22516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.039770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40898 (local address 127.3.217.3:44005)
0924 07:09:53.040540 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:53.040548 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:53.040565 (+ 17us) server_negotiation.cc:407] Connection header received
0924 07:09:53.040617 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:53.040621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:53.040743 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:53.040864 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:53.041078 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.042204 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.043801 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.044357 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.044465 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.044679 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:53.044754 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 07:09:53.044853 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":185,"threads_started":1}
I20260924 07:09:53.044981 22512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.038865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:44005 (local address 127.0.0.1:40898)
0924 07:09:53.039957 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:53.039971 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:53.040068 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:53.040860 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:53.040863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:53.040879 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:53.040965 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:53.040971 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.042384 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.042388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:53.043685 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:53.043694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.044477 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.044481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:53.044582 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.044619 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 07:09:53.044725 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 07:09:53.044836 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"mutex_wait_us":171,"thread_start_us":248,"threads_started":1}
I20260924 07:09:53.045290 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.038907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:39029 (local address 127.0.0.1:39034)
0924 07:09:53.040223 (+ 1316us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:53.040235 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:53.040309 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:53.041396 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:53.041399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:53.041415 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:53.041501 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:53.041507 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.042548 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.042551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:53.043820 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:53.043828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.044861 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.044865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:53.044947 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.044970 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:09:53.045015 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 07:09:53.045168 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1266,"mutex_wait_us":59,"thread_start_us":102,"threads_started":1}
I20260924 07:09:53.045656 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.038832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:42183 (local address 127.0.0.1:33504)
0924 07:09:53.039773 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:53.039823 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:53.039953 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:53.040761 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:53.040766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:53.040788 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:53.040919 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:53.040927 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.042365 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.042368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:53.043747 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:53.043757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.044994 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.044998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:53.045390 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.045413 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:09:53.045476 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 07:09:53.045541 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":72,"threads_started":1}
I20260924 07:09:53.045938 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.039675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39034 (local address 127.3.217.2:39029)
0924 07:09:53.041055 (+ 1380us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:53.041066 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:53.041084 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:09:53.041139 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:53.041142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:53.041271 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:53.041395 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:53.041590 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.042424 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.043930 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.044720 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.045531 (+ 811us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.045654 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:53.045703 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 07:09:53.045799 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1287,"thread_start_us":114,"threads_started":1}
I20260924 07:09:53.046090 22514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.038981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33504 (local address 127.3.217.1:42183)
0924 07:09:53.040286 (+ 1305us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:53.040302 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:53.040336 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:09:53.040418 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:53.040422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:53.040617 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:53.040761 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:53.041025 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.042200 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.043876 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.044720 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.044819 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.044944 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:53.045864 (+ 920us) server_negotiation.cc:299] Negotiation successful
0924 07:09:53.045954 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1193,"thread_start_us":104,"threads_started":1}
I20260924 07:09:53.048074 22300 tablet_service.cc:1467] Processing CreateTablet for tablet e44b6b12fa4d4632bb1cafd4be50ba93 (DEFAULT_TABLE table=test-table [id=979966c0c52048b296d8fcadd86d67c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:53.048056 22434 tablet_service.cc:1467] Processing CreateTablet for tablet e44b6b12fa4d4632bb1cafd4be50ba93 (DEFAULT_TABLE table=test-table [id=979966c0c52048b296d8fcadd86d67c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:53.048383 22166 tablet_service.cc:1467] Processing CreateTablet for tablet e44b6b12fa4d4632bb1cafd4be50ba93 (DEFAULT_TABLE table=test-table [id=979966c0c52048b296d8fcadd86d67c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:53.048967 22300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44b6b12fa4d4632bb1cafd4be50ba93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:53.049010 22434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44b6b12fa4d4632bb1cafd4be50ba93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:53.049208 22166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44b6b12fa4d4632bb1cafd4be50ba93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:53.054841 22517 tablet_bootstrap.cc:492] T e44b6b12fa4d4632bb1cafd4be50ba93 P 9b00a054d5a7497385ad15fc9f67361d: Bootstrap starting.
I20260924 07:09:53.056075 22518 tablet_bootstrap.cc:492] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0: Bootstrap starting.
I20260924 07:09:53.057559 22518 tablet_bootstrap.cc:654] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:53.057703 22517 tablet_bootstrap.cc:654] T e44b6b12fa4d4632bb1cafd4be50ba93 P 9b00a054d5a7497385ad15fc9f67361d: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:53.058351 22517 log.cc:824] T e44b6b12fa4d4632bb1cafd4be50ba93 P 9b00a054d5a7497385ad15fc9f67361d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:53.058491 22518 log.cc:824] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:53.059108 22519 tablet_bootstrap.cc:492] T e44b6b12fa4d4632bb1cafd4be50ba93 P 865ecf37641e4ea6a5a076870cbfcfa9: Bootstrap starting.
I20260924 07:09:53.060060 22517 tablet_bootstrap.cc:492] T e44b6b12fa4d4632bb1cafd4be50ba93 P 9b00a054d5a7497385ad15fc9f67361d: No bootstrap required, opened a new log
I20260924 07:09:53.060225 22517 ts_tablet_manager.cc:1397] T e44b6b12fa4d4632bb1cafd4be50ba93 P 9b00a054d5a7497385ad15fc9f67361d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:09:53.060273 22518 tablet_bootstrap.cc:492] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0: No bootstrap required, opened a new log
I20260924 07:09:53.060443 22518 ts_tablet_manager.cc:1397] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 07:09:53.061033 22519 tablet_bootstrap.cc:654] T e44b6b12fa4d4632bb1cafd4be50ba93 P 865ecf37641e4ea6a5a076870cbfcfa9: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:53.062080 22519 log.cc:824] T e44b6b12fa4d4632bb1cafd4be50ba93 P 865ecf37641e4ea6a5a076870cbfcfa9: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:53.063326 22517 raft_consensus.cc:348] T e44b6b12fa4d4632bb1cafd4be50ba93 P 9b00a054d5a7497385ad15fc9f67361d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ecf37641e4ea6a5a076870cbfcfa9" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 44005 } } peers { permanent_uuid: "483c017124c3480d9cdd1d570706f0f0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42183 } } peers { permanent_uuid: "9b00a054d5a7497385ad15fc9f67361d" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39029 } }
I20260924 07:09:53.063547 22517 raft_consensus.cc:374] T e44b6b12fa4d4632bb1cafd4be50ba93 P 9b00a054d5a7497385ad15fc9f67361d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:53.063599 22517 raft_consensus.cc:729] T e44b6b12fa4d4632bb1cafd4be50ba93 P 9b00a054d5a7497385ad15fc9f67361d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b00a054d5a7497385ad15fc9f67361d, State: Initialized, Role: FOLLOWER
I20260924 07:09:53.063959 22519 tablet_bootstrap.cc:492] T e44b6b12fa4d4632bb1cafd4be50ba93 P 865ecf37641e4ea6a5a076870cbfcfa9: No bootstrap required, opened a new log
I20260924 07:09:53.063975 22517 consensus_queue.cc:260] T e44b6b12fa4d4632bb1cafd4be50ba93 P 9b00a054d5a7497385ad15fc9f67361d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ecf37641e4ea6a5a076870cbfcfa9" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 44005 } } peers { permanent_uuid: "483c017124c3480d9cdd1d570706f0f0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42183 } } peers { permanent_uuid: "9b00a054d5a7497385ad15fc9f67361d" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39029 } }
I20260924 07:09:53.064143 22519 ts_tablet_manager.cc:1397] T e44b6b12fa4d4632bb1cafd4be50ba93 P 865ecf37641e4ea6a5a076870cbfcfa9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:09:53.063982 22518 raft_consensus.cc:348] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ecf37641e4ea6a5a076870cbfcfa9" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 44005 } } peers { permanent_uuid: "483c017124c3480d9cdd1d570706f0f0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42183 } } peers { permanent_uuid: "9b00a054d5a7497385ad15fc9f67361d" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39029 } }
I20260924 07:09:53.064206 22518 raft_consensus.cc:374] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:53.064258 22518 raft_consensus.cc:729] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 483c017124c3480d9cdd1d570706f0f0, State: Initialized, Role: FOLLOWER
I20260924 07:09:53.064672 22518 consensus_queue.cc:260] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ecf37641e4ea6a5a076870cbfcfa9" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 44005 } } peers { permanent_uuid: "483c017124c3480d9cdd1d570706f0f0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42183 } } peers { permanent_uuid: "9b00a054d5a7497385ad15fc9f67361d" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39029 } }
I20260924 07:09:53.064854 22517 ts_tablet_manager.cc:1428] T e44b6b12fa4d4632bb1cafd4be50ba93 P 9b00a054d5a7497385ad15fc9f67361d: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260924 07:09:53.065182 22366 heartbeater.cc:503] Master 127.3.217.62:41751 was elected leader, sending a full tablet report...
I20260924 07:09:53.065524 22518 ts_tablet_manager.cc:1428] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 07:09:53.066103 22232 heartbeater.cc:503] Master 127.3.217.62:41751 was elected leader, sending a full tablet report...
I20260924 07:09:53.069092 22519 raft_consensus.cc:348] T e44b6b12fa4d4632bb1cafd4be50ba93 P 865ecf37641e4ea6a5a076870cbfcfa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ecf37641e4ea6a5a076870cbfcfa9" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 44005 } } peers { permanent_uuid: "483c017124c3480d9cdd1d570706f0f0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42183 } } peers { permanent_uuid: "9b00a054d5a7497385ad15fc9f67361d" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39029 } }
I20260924 07:09:53.069293 22519 raft_consensus.cc:374] T e44b6b12fa4d4632bb1cafd4be50ba93 P 865ecf37641e4ea6a5a076870cbfcfa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:53.069334 22519 raft_consensus.cc:729] T e44b6b12fa4d4632bb1cafd4be50ba93 P 865ecf37641e4ea6a5a076870cbfcfa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865ecf37641e4ea6a5a076870cbfcfa9, State: Initialized, Role: FOLLOWER
I20260924 07:09:53.069664 22519 consensus_queue.cc:260] T e44b6b12fa4d4632bb1cafd4be50ba93 P 865ecf37641e4ea6a5a076870cbfcfa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ecf37641e4ea6a5a076870cbfcfa9" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 44005 } } peers { permanent_uuid: "483c017124c3480d9cdd1d570706f0f0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42183 } } peers { permanent_uuid: "9b00a054d5a7497385ad15fc9f67361d" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39029 } }
I20260924 07:09:53.070205 22500 heartbeater.cc:503] Master 127.3.217.62:41751 was elected leader, sending a full tablet report...
I20260924 07:09:53.070428 22519 ts_tablet_manager.cc:1428] T e44b6b12fa4d4632bb1cafd4be50ba93 P 865ecf37641e4ea6a5a076870cbfcfa9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260924 07:09:53.072006 22233 tablet.cc:2369] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:09:53.093976 22524 raft_consensus.cc:482] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:09:53.094187 22524 raft_consensus.cc:504] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ecf37641e4ea6a5a076870cbfcfa9" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 44005 } } peers { permanent_uuid: "483c017124c3480d9cdd1d570706f0f0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42183 } } peers { permanent_uuid: "9b00a054d5a7497385ad15fc9f67361d" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39029 } }
I20260924 07:09:53.095237 22524 leader_election.cc:288] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 865ecf37641e4ea6a5a076870cbfcfa9 (127.3.217.3:44005), 9b00a054d5a7497385ad15fc9f67361d (127.3.217.2:39029)
I20260924 07:09:53.100250 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.095483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:44005 (local address 127.3.217.1:40719)
0924 07:09:53.095916 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:53.095926 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:53.095996 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:53.096226 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:53.096229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:53.096266 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:53.096461 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:53.096469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.097461 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.097465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:53.098810 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:53.098816 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.099836 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.099840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:53.099937 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.099955 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 07:09:53.100021 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 07:09:53.100095 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":140,"spinlock_wait_cycles":35456,"thread_start_us":86,"threads_started":1}
I20260924 07:09:53.100289 22516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.095606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:40719 (local address 127.3.217.3:44005)
0924 07:09:53.095726 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:53.095731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:53.095986 (+ 255us) server_negotiation.cc:407] Connection header received
0924 07:09:53.096094 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:53.096097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:53.096138 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:53.096201 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:53.096567 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.097336 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.098941 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.099567 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.099672 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.099732 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:53.100109 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 07:09:53.100180 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:09:53.100401 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.095618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:35835 (local address 127.3.217.2:39029)
0924 07:09:53.095726 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:53.095736 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:53.095933 (+ 197us) server_negotiation.cc:407] Connection header received
0924 07:09:53.096075 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:53.096078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:53.096124 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:53.096202 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:53.096567 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.097344 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.099077 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.099711 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.099803 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.099854 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:53.100219 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 07:09:53.100287 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:09:53.100533 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.095482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:39029 (local address 127.3.217.1:35835)
0924 07:09:53.095847 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:53.095862 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:53.095975 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:53.096226 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:53.096229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:53.096267 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:53.096474 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:53.096479 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.097461 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.097465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:53.098804 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:53.098816 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.099951 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.099968 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:53.100088 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.100110 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:09:53.100180 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 07:09:53.100251 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":122,"spinlock_wait_cycles":5504,"thread_start_us":108,"threads_started":1}
I20260924 07:09:53.100804 22454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44b6b12fa4d4632bb1cafd4be50ba93" candidate_uuid: "483c017124c3480d9cdd1d570706f0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865ecf37641e4ea6a5a076870cbfcfa9" is_pre_election: true
I20260924 07:09:53.101028 22320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44b6b12fa4d4632bb1cafd4be50ba93" candidate_uuid: "483c017124c3480d9cdd1d570706f0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b00a054d5a7497385ad15fc9f67361d" is_pre_election: true
I20260924 07:09:53.101060 22454 raft_consensus.cc:2454] T e44b6b12fa4d4632bb1cafd4be50ba93 P 865ecf37641e4ea6a5a076870cbfcfa9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 483c017124c3480d9cdd1d570706f0f0 in term 0.
I20260924 07:09:53.101258 22320 raft_consensus.cc:2454] T e44b6b12fa4d4632bb1cafd4be50ba93 P 9b00a054d5a7497385ad15fc9f67361d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 483c017124c3480d9cdd1d570706f0f0 in term 0.
I20260924 07:09:53.101528 22122 leader_election.cc:302] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 483c017124c3480d9cdd1d570706f0f0, 865ecf37641e4ea6a5a076870cbfcfa9; no voters:
I20260924 07:09:53.101769 22524 raft_consensus.cc:2781] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:09:53.101852 22524 raft_consensus.cc:482] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:09:53.101893 22524 raft_consensus.cc:3037] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:53.102852 22524 raft_consensus.cc:504] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ecf37641e4ea6a5a076870cbfcfa9" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 44005 } } peers { permanent_uuid: "483c017124c3480d9cdd1d570706f0f0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42183 } } peers { permanent_uuid: "9b00a054d5a7497385ad15fc9f67361d" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39029 } }
I20260924 07:09:53.103152 22524 leader_election.cc:288] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0 [CANDIDATE]: Term 1 election: Requested vote from peers 865ecf37641e4ea6a5a076870cbfcfa9 (127.3.217.3:44005), 9b00a054d5a7497385ad15fc9f67361d (127.3.217.2:39029)
I20260924 07:09:53.103439 22454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44b6b12fa4d4632bb1cafd4be50ba93" candidate_uuid: "483c017124c3480d9cdd1d570706f0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865ecf37641e4ea6a5a076870cbfcfa9"
I20260924 07:09:53.103564 22454 raft_consensus.cc:3037] T e44b6b12fa4d4632bb1cafd4be50ba93 P 865ecf37641e4ea6a5a076870cbfcfa9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:53.103552 22320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44b6b12fa4d4632bb1cafd4be50ba93" candidate_uuid: "483c017124c3480d9cdd1d570706f0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b00a054d5a7497385ad15fc9f67361d"
I20260924 07:09:53.103688 22320 raft_consensus.cc:3037] T e44b6b12fa4d4632bb1cafd4be50ba93 P 9b00a054d5a7497385ad15fc9f67361d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:53.104521 22454 raft_consensus.cc:2454] T e44b6b12fa4d4632bb1cafd4be50ba93 P 865ecf37641e4ea6a5a076870cbfcfa9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 483c017124c3480d9cdd1d570706f0f0 in term 1.
I20260924 07:09:53.104724 22320 raft_consensus.cc:2454] T e44b6b12fa4d4632bb1cafd4be50ba93 P 9b00a054d5a7497385ad15fc9f67361d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 483c017124c3480d9cdd1d570706f0f0 in term 1.
I20260924 07:09:53.104776 22122 leader_election.cc:302] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 483c017124c3480d9cdd1d570706f0f0, 865ecf37641e4ea6a5a076870cbfcfa9; no voters:
I20260924 07:09:53.104935 22524 raft_consensus.cc:2781] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:53.105115 22524 raft_consensus.cc:686] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0 [term 1 LEADER]: Becoming Leader. State: Replica: 483c017124c3480d9cdd1d570706f0f0, State: Running, Role: LEADER
I20260924 07:09:53.105417 22524 consensus_queue.cc:237] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ecf37641e4ea6a5a076870cbfcfa9" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 44005 } } peers { permanent_uuid: "483c017124c3480d9cdd1d570706f0f0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42183 } } peers { permanent_uuid: "9b00a054d5a7497385ad15fc9f67361d" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39029 } }
I20260924 07:09:53.108414 22039 catalog_manager.cc:5495] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 483c017124c3480d9cdd1d570706f0f0 (127.3.217.1). New cstate: current_term: 1 leader_uuid: "483c017124c3480d9cdd1d570706f0f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ecf37641e4ea6a5a076870cbfcfa9" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 44005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "483c017124c3480d9cdd1d570706f0f0" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b00a054d5a7497385ad15fc9f67361d" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39029 } health_report { overall_health: UNKNOWN } } }
I20260924 07:09:53.124171 22514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.120399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33514 (local address 127.3.217.1:42183)
0924 07:09:53.120682 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:53.120686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:53.120713 (+ 27us) server_negotiation.cc:407] Connection header received
0924 07:09:53.120824 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:53.120828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:53.120916 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:53.121136 (+ 220us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:09:53.121331 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.121824 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.122645 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.122857 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.122940 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.123300 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:09:53.123752 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:09:53.123878 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:53.124000 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 07:09:53.124037 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260924 07:09:53.129333 22320 raft_consensus.cc:1261] T e44b6b12fa4d4632bb1cafd4be50ba93 P 9b00a054d5a7497385ad15fc9f67361d [term 1 FOLLOWER]: Refusing update from remote peer 483c017124c3480d9cdd1d570706f0f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:53.129591 22454 raft_consensus.cc:1261] T e44b6b12fa4d4632bb1cafd4be50ba93 P 865ecf37641e4ea6a5a076870cbfcfa9 [term 1 FOLLOWER]: Refusing update from remote peer 483c017124c3480d9cdd1d570706f0f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:53.130017 22528 consensus_queue.cc:1035] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b00a054d5a7497385ad15fc9f67361d" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:53.130205 22524 consensus_queue.cc:1035] T e44b6b12fa4d4632bb1cafd4be50ba93 P 483c017124c3480d9cdd1d570706f0f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "865ecf37641e4ea6a5a076870cbfcfa9" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:53.144162 22040 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:53.144322 22040 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:53.146502 22040 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1cf1723ecfd4ec0a5675a7a9b8faf72: Sending DeleteTablet for 3 replicas of tablet e44b6b12fa4d4632bb1cafd4be50ba93
I20260924 07:09:53.147245 22166 tablet_service.cc:1514] Processing DeleteTablet for tablet e44b6b12fa4d4632bb1cafd4be50ba93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:53 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:33504
I20260924 07:09:53.147320 22434 tablet_service.cc:1514] Processing DeleteTablet for tablet e44b6b12fa4d4632bb1cafd4be50ba93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:53 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:40898
I20260924 07:09:53.147421 22300 tablet_service.cc:1514] Processing DeleteTablet for tablet e44b6b12fa4d4632bb1cafd4be50ba93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:53 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:39034
I20260924 07:09:53.147890 22542 tablet_replica.cc:330] T e44b6b12fa4d4632bb1cafd4be50ba93 P 9b00a054d5a7497385ad15fc9f67361d: stopping tablet replica
I20260924 07:09:53.147938 22541 tablet_replica.cc:330] T e44b6b12fa4d4632bb1cafd4be50ba93 P 865ecf37641e4ea6a5a076870cbfcfa9: stopping tablet replica
I20260924 07:09:53.148171 22542 raft_consensus.cc:2229] T e44b6b12fa4d4632bb1cafd4be50ba93 P 9b00a054d5a7497385ad15fc9f67361d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:53.148298 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 22103
I20260924 07:09:53.148396 22541 raft_consensus.cc:2229] T e44b6b12fa4d4632bb1cafd4be50ba93 P 865ecf37641e4ea6a5a076870cbfcfa9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:53.148454 22542 raft_consensus.cc:2258] T e44b6b12fa4d4632bb1cafd4be50ba93 P 9b00a054d5a7497385ad15fc9f67361d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:53.148938 22541 raft_consensus.cc:2258] T e44b6b12fa4d4632bb1cafd4be50ba93 P 865ecf37641e4ea6a5a076870cbfcfa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:53.150249 22542 ts_tablet_manager.cc:1905] T e44b6b12fa4d4632bb1cafd4be50ba93 P 9b00a054d5a7497385ad15fc9f67361d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:53.150897 22541 ts_tablet_manager.cc:1905] T e44b6b12fa4d4632bb1cafd4be50ba93 P 865ecf37641e4ea6a5a076870cbfcfa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:53.152997 22541 ts_tablet_manager.cc:1918] T e44b6b12fa4d4632bb1cafd4be50ba93 P 865ecf37641e4ea6a5a076870cbfcfa9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:09:53.153076 22541 log.cc:1196] T e44b6b12fa4d4632bb1cafd4be50ba93 P 865ecf37641e4ea6a5a076870cbfcfa9: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/e44b6b12fa4d4632bb1cafd4be50ba93
I20260924 07:09:53.153090 22542 ts_tablet_manager.cc:1918] T e44b6b12fa4d4632bb1cafd4be50ba93 P 9b00a054d5a7497385ad15fc9f67361d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:09:53.153165 22542 log.cc:1196] T e44b6b12fa4d4632bb1cafd4be50ba93 P 9b00a054d5a7497385ad15fc9f67361d: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/e44b6b12fa4d4632bb1cafd4be50ba93
I20260924 07:09:53.153371 22541 ts_tablet_manager.cc:1939] T e44b6b12fa4d4632bb1cafd4be50ba93 P 865ecf37641e4ea6a5a076870cbfcfa9: Deleting consensus metadata
I20260924 07:09:53.153555 22542 ts_tablet_manager.cc:1939] T e44b6b12fa4d4632bb1cafd4be50ba93 P 9b00a054d5a7497385ad15fc9f67361d: Deleting consensus metadata
I20260924 07:09:53.154551 22028 catalog_manager.cc:4841] TS 865ecf37641e4ea6a5a076870cbfcfa9 (127.3.217.3:44005): tablet e44b6b12fa4d4632bb1cafd4be50ba93 (table test-table [id=979966c0c52048b296d8fcadd86d67c0]) successfully deleted
I20260924 07:09:53.154678 22026 catalog_manager.cc:4841] TS 9b00a054d5a7497385ad15fc9f67361d (127.3.217.2:39029): tablet e44b6b12fa4d4632bb1cafd4be50ba93 (table test-table [id=979966c0c52048b296d8fcadd86d67c0]) successfully deleted
W20260924 07:09:53.156052 22025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.217.1:42183 (error 108)
I20260924 07:09:53.156083 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 22237
I20260924 07:09:53.157419 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.157161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:42183 (local address 127.0.0.1:33520)
0924 07:09:53.157281 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:53.157343 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.1:42183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 07:09:53.157567 22025 catalog_manager.cc:4584] TS 483c017124c3480d9cdd1d570706f0f0 (127.3.217.1:42183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e44b6b12fa4d4632bb1cafd4be50ba93: Network error: Client connection negotiation failed: client connection to 127.3.217.1:42183: connect: Connection refused (error 111)
I20260924 07:09:53.161016 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 22371
I20260924 07:09:53.165853 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 22009
2026-09-24T07:09:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2127 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T07:09:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:09:53Z Disabled control of system clock
I20260924 07:09:53.194146 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:42511
--webserver_interface=127.3.217.62
--webserver_port=0
--builtin_ntp_servers=127.3.217.20:36733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:42511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:09:53.302481 22547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:53.302711 22547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:53.302752 22547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:53.306663 22547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:09:53.306715 22547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:53.306730 22547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:53.306751 22547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:09:53.306766 22547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:09:53.311385 22547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:36733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:42511
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:42511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:53.312355 22547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:53.313563 22547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:53.321456 22552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:53.321468 22553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:53.321513 22547 server_base.cc:1034] running on GCE node
W20260924 07:09:53.321506 22555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:53.322436 22547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:53.323439 22547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:53.324641 22547 hybrid_clock.cc:648] HybridClock initialized: now 1790233793324606 us; error 70 us; skew 500 ppm
I20260924 07:09:53.326500 22547 webserver.cc:460] Webserver started at http://127.3.217.62:45699/ using document root <none> and password file <none>
I20260924 07:09:53.327014 22547 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:53.327077 22547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:53.327250 22547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:53.329214 22547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "687b1a4b59ff4ab6b2349dcf0aaba042"
format_stamp: "Formatted at 2026-09-24 07:09:53 on dist-test-slave-2p7j"
server_key: "1014dfa7c308f0b073abef51a4f67d8a"
server_key_iv: "47f25aeff75f83b17e53dd2fdfd46c89"
server_key_version: "encryptionkey@0"
I20260924 07:09:53.329730 22547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "687b1a4b59ff4ab6b2349dcf0aaba042"
format_stamp: "Formatted at 2026-09-24 07:09:53 on dist-test-slave-2p7j"
server_key: "1014dfa7c308f0b073abef51a4f67d8a"
server_key_iv: "47f25aeff75f83b17e53dd2fdfd46c89"
server_key_version: "encryptionkey@0"
I20260924 07:09:53.333449 22547 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:09:53.335878 22561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:53.337138 22547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:09:53.337256 22547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "687b1a4b59ff4ab6b2349dcf0aaba042"
format_stamp: "Formatted at 2026-09-24 07:09:53 on dist-test-slave-2p7j"
server_key: "1014dfa7c308f0b073abef51a4f67d8a"
server_key_iv: "47f25aeff75f83b17e53dd2fdfd46c89"
server_key_version: "encryptionkey@0"
I20260924 07:09:53.337352 22547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:53.366349 22547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:53.367007 22547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:53.367180 22547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:53.374183 22547 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:42511
I20260924 07:09:53.374359 22613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:42511 every 8 connection(s)
I20260924 07:09:53.375098 22547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:09:53.379002 22614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:53.380762 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 22547
I20260924 07:09:53.380928 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:09:53.381214 3940 external_mini_cluster.cc:1428] Setting key 3a3ef58de922da9a5981c57b8edc57a0
I20260924 07:09:53.384819 22614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 687b1a4b59ff4ab6b2349dcf0aaba042: Bootstrap starting.
I20260924 07:09:53.387005 22614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 687b1a4b59ff4ab6b2349dcf0aaba042: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:53.387995 22614 log.cc:824] T 00000000000000000000000000000000 P 687b1a4b59ff4ab6b2349dcf0aaba042: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:53.390038 22614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 687b1a4b59ff4ab6b2349dcf0aaba042: No bootstrap required, opened a new log
I20260924 07:09:53.391464 22617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.383053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50542 (local address 127.3.217.62:42511)
0924 07:09:53.383875 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:53.383892 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:53.383926 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:09:53.384631 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:53.384649 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:53.385045 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:53.385313 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:53.385678 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.386642 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.387953 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.388795 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.388927 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.389780 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:53.389785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:53.389796 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:53.390178 (+ 382us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:53.390237 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:53.390258 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:53.390331 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:53.390525 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 07:09:53.390800 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":155,"threads_started":1}
I20260924 07:09:53.394419 22614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 687b1a4b59ff4ab6b2349dcf0aaba042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687b1a4b59ff4ab6b2349dcf0aaba042" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42511 } }
I20260924 07:09:53.394659 22614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 687b1a4b59ff4ab6b2349dcf0aaba042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:53.394699 22614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 687b1a4b59ff4ab6b2349dcf0aaba042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 687b1a4b59ff4ab6b2349dcf0aaba042, State: Initialized, Role: FOLLOWER
I20260924 07:09:53.395097 22614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 687b1a4b59ff4ab6b2349dcf0aaba042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687b1a4b59ff4ab6b2349dcf0aaba042" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42511 } }
I20260924 07:09:53.395236 22614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 687b1a4b59ff4ab6b2349dcf0aaba042 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:09:53.395284 22614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 687b1a4b59ff4ab6b2349dcf0aaba042 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:09:53.395381 22614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 687b1a4b59ff4ab6b2349dcf0aaba042 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:53.396725 22614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 687b1a4b59ff4ab6b2349dcf0aaba042 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687b1a4b59ff4ab6b2349dcf0aaba042" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42511 } }
I20260924 07:09:53.397053 22614 leader_election.cc:302] T 00000000000000000000000000000000 P 687b1a4b59ff4ab6b2349dcf0aaba042 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 687b1a4b59ff4ab6b2349dcf0aaba042; no voters:
I20260924 07:09:53.397596 22614 leader_election.cc:288] T 00000000000000000000000000000000 P 687b1a4b59ff4ab6b2349dcf0aaba042 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:09:53.397827 22619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 687b1a4b59ff4ab6b2349dcf0aaba042 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:53.398119 22619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 687b1a4b59ff4ab6b2349dcf0aaba042 [term 1 LEADER]: Becoming Leader. State: Replica: 687b1a4b59ff4ab6b2349dcf0aaba042, State: Running, Role: LEADER
I20260924 07:09:53.398437 22619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 687b1a4b59ff4ab6b2349dcf0aaba042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687b1a4b59ff4ab6b2349dcf0aaba042" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42511 } }
I20260924 07:09:53.399206 22614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 687b1a4b59ff4ab6b2349dcf0aaba042 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:09:53.399995 22620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 687b1a4b59ff4ab6b2349dcf0aaba042 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "687b1a4b59ff4ab6b2349dcf0aaba042" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687b1a4b59ff4ab6b2349dcf0aaba042" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42511 } } }
I20260924 07:09:53.400120 22620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 687b1a4b59ff4ab6b2349dcf0aaba042 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:53.400511 22621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 687b1a4b59ff4ab6b2349dcf0aaba042 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 687b1a4b59ff4ab6b2349dcf0aaba042. Latest consensus state: current_term: 1 leader_uuid: "687b1a4b59ff4ab6b2349dcf0aaba042" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687b1a4b59ff4ab6b2349dcf0aaba042" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42511 } } }
I20260924 07:09:53.400978 22621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 687b1a4b59ff4ab6b2349dcf0aaba042 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:53.401430 22624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:09:53.404085 22624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:09:53.411342 22624 catalog_manager.cc:1347] Generated new cluster ID: 2e6aecbdeec04227b5e1ace9c33c5f39
I20260924 07:09:53.411410 22624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:09:53.439970 22624 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:09:53.441406 22624 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:09:53.447870 22624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 687b1a4b59ff4ab6b2349dcf0aaba042: Generated new TSK 0
I20260924 07:09:53.448485 22624 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:09:53.454130 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42511
--builtin_ntp_servers=127.3.217.20:36733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:09:53.583892 22638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:53.584129 22638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:53.584165 22638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:53.588090 22638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:53.588141 22638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:53.588194 22638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:09:53.594223 22638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:36733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:53.595780 22638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:53.597240 22638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:53.607201 22643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:53.607270 22646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:53.607201 22644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:53.607617 22638 server_base.cc:1034] running on GCE node
I20260924 07:09:53.608029 22638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:53.608637 22638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:53.609817 22638 hybrid_clock.cc:648] HybridClock initialized: now 1790233793609799 us; error 31 us; skew 500 ppm
I20260924 07:09:53.611591 22638 webserver.cc:460] Webserver started at http://127.3.217.1:45419/ using document root <none> and password file <none>
I20260924 07:09:53.612119 22638 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:53.612178 22638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:53.612460 22638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:53.614498 22638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "1dd0fb0ae21c472f92ba31a119a83886"
format_stamp: "Formatted at 2026-09-24 07:09:53 on dist-test-slave-2p7j"
server_key: "d302ae21773cf1eba33483c2432394b4"
server_key_iv: "704a99efe676eee247c9457d3a1d5b21"
server_key_version: "encryptionkey@0"
I20260924 07:09:53.615042 22638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "1dd0fb0ae21c472f92ba31a119a83886"
format_stamp: "Formatted at 2026-09-24 07:09:53 on dist-test-slave-2p7j"
server_key: "d302ae21773cf1eba33483c2432394b4"
server_key_iv: "704a99efe676eee247c9457d3a1d5b21"
server_key_version: "encryptionkey@0"
I20260924 07:09:53.618865 22638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260924 07:09:53.621716 22652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:53.622965 22638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 07:09:53.623065 22638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "1dd0fb0ae21c472f92ba31a119a83886"
format_stamp: "Formatted at 2026-09-24 07:09:53 on dist-test-slave-2p7j"
server_key: "d302ae21773cf1eba33483c2432394b4"
server_key_iv: "704a99efe676eee247c9457d3a1d5b21"
server_key_version: "encryptionkey@0"
I20260924 07:09:53.623178 22638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:53.632721 22638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:53.633440 22638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:53.633599 22638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:53.634449 22638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:53.635452 22638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:53.635550 22638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:53.635613 22638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:53.635653 22638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:53.644704 22638 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:39429
I20260924 07:09:53.644773 22765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:39429 every 8 connection(s)
I20260924 07:09:53.645573 22638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:09:53.652125 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 22638
I20260924 07:09:53.652269 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:09:53.652545 3940 external_mini_cluster.cc:1428] Setting key f928840b5d16dbc1891ea9e86909be9e
I20260924 07:09:53.654336 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42511
--builtin_ntp_servers=127.3.217.20:36733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 07:09:53.655977 22617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.647895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:52187 (local address 127.3.217.62:42511)
0924 07:09:53.648056 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:53.648061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:53.649411 (+ 1350us) server_negotiation.cc:407] Connection header received
0924 07:09:53.650844 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:53.650847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:53.650893 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:53.650970 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:53.651907 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.652703 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.653856 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.654167 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.654274 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.655456 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:53.655460 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:53.655462 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:53.655475 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:53.655526 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:53.655537 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:53.655609 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:53.655815 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 07:09:53.655870 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 07:09:53.657015 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.648303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42511 (local address 127.3.217.1:52187)
0924 07:09:53.649193 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:53.649258 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:53.650616 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:53.651155 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:53.651161 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:53.651242 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:53.651759 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:53.651774 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.652848 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.652851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:53.653755 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:53.653765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.654336 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.654339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:53.654526 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.655200 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:09:53.655201 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:53.655241 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:09:53.655303 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:53.655615 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:53.655624 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:53.655631 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 07:09:53.655776 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 07:09:53.656062 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":117,"threads_started":1}
I20260924 07:09:53.658227 22766 heartbeater.cc:347] Connected to a master server at 127.3.217.62:42511
I20260924 07:09:53.658439 22766 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:53.658910 22766 heartbeater.cc:511] Master 127.3.217.62:42511 requested a full tablet report, sending...
I20260924 07:09:53.660346 22578 ts_manager.cc:194] Registered new tserver with Master: 1dd0fb0ae21c472f92ba31a119a83886 (127.3.217.1:39429)
I20260924 07:09:53.661409 22578 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.217.1:52187
W20260924 07:09:53.769536 22769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:53.769793 22769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:53.769860 22769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:53.774487 22769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:53.774540 22769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:53.774633 22769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:09:53.779371 22769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:36733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:53.780421 22769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:53.781792 22769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:53.791890 22774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:53.791896 22777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:53.791896 22775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:53.792114 22769 server_base.cc:1034] running on GCE node
I20260924 07:09:53.792629 22769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:53.793201 22769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:53.794380 22769 hybrid_clock.cc:648] HybridClock initialized: now 1790233793794354 us; error 39 us; skew 500 ppm
I20260924 07:09:53.796288 22769 webserver.cc:460] Webserver started at http://127.3.217.2:34857/ using document root <none> and password file <none>
I20260924 07:09:53.796833 22769 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:53.796904 22769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:53.797089 22769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:53.799093 22769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "b89905195e024ce09707c1f4a5d62271"
format_stamp: "Formatted at 2026-09-24 07:09:53 on dist-test-slave-2p7j"
server_key: "ccfd7a31654b64f676142972d955f1b3"
server_key_iv: "12abfd4a9822679f9cf93822b7f5ba44"
server_key_version: "encryptionkey@0"
I20260924 07:09:53.799676 22769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "b89905195e024ce09707c1f4a5d62271"
format_stamp: "Formatted at 2026-09-24 07:09:53 on dist-test-slave-2p7j"
server_key: "ccfd7a31654b64f676142972d955f1b3"
server_key_iv: "12abfd4a9822679f9cf93822b7f5ba44"
server_key_version: "encryptionkey@0"
I20260924 07:09:53.803676 22769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:09:53.806437 22783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:53.807606 22769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:09:53.807711 22769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "b89905195e024ce09707c1f4a5d62271"
format_stamp: "Formatted at 2026-09-24 07:09:53 on dist-test-slave-2p7j"
server_key: "ccfd7a31654b64f676142972d955f1b3"
server_key_iv: "12abfd4a9822679f9cf93822b7f5ba44"
server_key_version: "encryptionkey@0"
I20260924 07:09:53.807809 22769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:53.829859 22769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:53.830489 22769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:53.830648 22769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:53.831205 22769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:53.832094 22769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:53.832144 22769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:53.832206 22769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:53.832232 22769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:53.841642 22769 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:39687
I20260924 07:09:53.841676 22896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:39687 every 8 connection(s)
I20260924 07:09:53.842463 22769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:09:53.850903 22617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.844164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:40687 (local address 127.3.217.62:42511)
0924 07:09:53.844296 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:53.844300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:53.845336 (+ 1036us) server_negotiation.cc:407] Connection header received
0924 07:09:53.846610 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:53.846612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:53.846655 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:53.846723 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:53.847542 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.848117 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.849206 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.849413 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.849489 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.850486 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:53.850489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:53.850491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:53.850502 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:53.850520 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:53.850527 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:53.850588 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:53.850748 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 07:09:53.850795 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:09:53.851455 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 22769
I20260924 07:09:53.851595 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:09:53.851902 3940 external_mini_cluster.cc:1428] Setting key e6d7501b4f614edc5c3e0358f37fdb99
I20260924 07:09:53.852434 22899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.844545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42511 (local address 127.3.217.2:40687)
0924 07:09:53.845211 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:53.845248 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:53.846433 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:53.846881 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:53.846887 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:53.846959 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:53.847392 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:53.847403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.848242 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.848245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:53.849094 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:53.849103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.849561 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.849565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:53.849730 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.850265 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:09:53.850267 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:53.850301 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:09:53.850355 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:53.850624 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:53.850632 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:53.850637 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:09:53.850761 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 07:09:53.851256 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":110,"threads_started":1}
I20260924 07:09:53.853570 22897 heartbeater.cc:347] Connected to a master server at 127.3.217.62:42511
I20260924 07:09:53.853776 22897 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:53.853825 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42511
--builtin_ntp_servers=127.3.217.20:36733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 07:09:53.854252 22897 heartbeater.cc:511] Master 127.3.217.62:42511 requested a full tablet report, sending...
I20260924 07:09:53.855230 22577 ts_manager.cc:194] Registered new tserver with Master: b89905195e024ce09707c1f4a5d62271 (127.3.217.2:39687)
I20260924 07:09:53.855800 22577 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.217.2:40687
W20260924 07:09:53.963356 22900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:53.963642 22900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:53.963683 22900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:53.967589 22900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:53.967635 22900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:53.967687 22900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:09:53.972294 22900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:36733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:42511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:53.973206 22900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:53.974433 22900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:53.983840 22906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:53.983923 22905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:53.983968 22908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:53.984018 22900 server_base.cc:1034] running on GCE node
I20260924 07:09:53.984472 22900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:53.985141 22900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:53.986335 22900 hybrid_clock.cc:648] HybridClock initialized: now 1790233793986309 us; error 44 us; skew 500 ppm
I20260924 07:09:53.988049 22900 webserver.cc:460] Webserver started at http://127.3.217.3:40117/ using document root <none> and password file <none>
I20260924 07:09:53.988631 22900 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:53.988701 22900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:53.988898 22900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:53.990773 22900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "8ca41da710324f18b691e51f94e8358e"
format_stamp: "Formatted at 2026-09-24 07:09:53 on dist-test-slave-2p7j"
server_key: "5d46f8450ac9fe0662fbf315d23bea16"
server_key_iv: "f9509bb52404300fc5be5a8ea1443c75"
server_key_version: "encryptionkey@0"
I20260924 07:09:53.991250 22900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "8ca41da710324f18b691e51f94e8358e"
format_stamp: "Formatted at 2026-09-24 07:09:53 on dist-test-slave-2p7j"
server_key: "5d46f8450ac9fe0662fbf315d23bea16"
server_key_iv: "f9509bb52404300fc5be5a8ea1443c75"
server_key_version: "encryptionkey@0"
I20260924 07:09:53.994913 22900 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 07:09:53.997437 22914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:53.998847 22900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260924 07:09:53.998947 22900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "8ca41da710324f18b691e51f94e8358e"
format_stamp: "Formatted at 2026-09-24 07:09:53 on dist-test-slave-2p7j"
server_key: "5d46f8450ac9fe0662fbf315d23bea16"
server_key_iv: "f9509bb52404300fc5be5a8ea1443c75"
server_key_version: "encryptionkey@0"
I20260924 07:09:53.999037 22900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:54.013034 22900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:54.013566 22900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:54.013698 22900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:54.014154 22900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:54.014950 22900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:54.014994 22900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:54.015029 22900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:54.015049 22900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:54.023571 22900 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:35469
I20260924 07:09:54.023617 23027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:35469 every 8 connection(s)
I20260924 07:09:54.024477 22900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:09:54.029665 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 22900
I20260924 07:09:54.029784 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:09:54.030025 3940 external_mini_cluster.cc:1428] Setting key 776cd26f20e3d42c48d1d93ff811c03c
I20260924 07:09:54.033569 22617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:54.026367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:55731 (local address 127.3.217.62:42511)
0924 07:09:54.026492 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:54.026496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:54.027481 (+ 985us) server_negotiation.cc:407] Connection header received
0924 07:09:54.028831 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:54.028834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:54.028878 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:54.028952 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:54.029843 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.030552 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.031791 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.032051 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.032144 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:54.033133 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:54.033136 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:54.033139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:54.033151 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:54.033172 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:54.033179 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:54.033250 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:54.033373 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 07:09:54.033428 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:09:54.034312 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:54.026734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42511 (local address 127.3.217.3:55731)
0924 07:09:54.027345 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:54.027381 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:54.028632 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:54.029142 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:54.029148 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:54.029237 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:54.029695 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:54.029708 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.030669 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.030672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:54.031685 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:54.031696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.032188 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.032191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:54.032342 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:54.032887 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:09:54.032889 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:54.032926 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:09:54.032991 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:54.033248 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:54.033256 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:54.033261 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:09:54.033366 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 07:09:54.033625 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":108,"threads_started":1}
I20260924 07:09:54.035737 23028 heartbeater.cc:347] Connected to a master server at 127.3.217.62:42511
I20260924 07:09:54.035951 23028 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:54.036742 23028 heartbeater.cc:511] Master 127.3.217.62:42511 requested a full tablet report, sending...
I20260924 07:09:54.038122 22577 ts_manager.cc:194] Registered new tserver with Master: 8ca41da710324f18b691e51f94e8358e (127.3.217.3:35469)
I20260924 07:09:54.038852 22577 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.217.3:55731
I20260924 07:09:54.043661 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:09:54.049291 22617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:54.045655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50544 (local address 127.3.217.62:42511)
0924 07:09:54.045795 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:54.045799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:54.045968 (+ 169us) server_negotiation.cc:407] Connection header received
0924 07:09:54.046086 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:54.046089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:54.046128 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:54.046187 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:54.046454 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.046947 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.047816 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.048023 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.048121 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:54.048868 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:54.048870 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:54.048872 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:54.048883 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:54.048900 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:54.048907 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:54.048995 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:54.049126 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 07:09:54.049163 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:09:54.052019 22577 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:09:54.054955 22577 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:09:54.069463 23043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:54.064549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46618 (local address 127.3.217.1:39429)
0924 07:09:54.064915 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:54.064926 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:54.064950 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:09:54.065025 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:54.065030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:54.065226 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:54.065698 (+ 472us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:54.065930 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.066618 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.068208 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.068829 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.068930 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:54.069118 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:54.069219 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 07:09:54.069318 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":94,"threads_started":1}
I20260924 07:09:54.069459 23039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:54.063850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:39429 (local address 127.0.0.1:46618)
0924 07:09:54.064484 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:54.064533 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:54.064658 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:54.065395 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:54.065401 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:54.065423 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:54.065559 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:54.065567 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.066724 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.066728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:54.068077 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:54.068088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.068942 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.068946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:54.069050 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:54.069084 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 07:09:54.069190 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 07:09:54.069309 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":22,"spinlock_wait_cycles":5760,"thread_start_us":95,"threads_started":1}
I20260924 07:09:54.069695 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:54.064069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:39687 (local address 127.0.0.1:46960)
0924 07:09:54.065075 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:54.065087 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:54.065173 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:54.065612 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:54.065614 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:54.065632 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:54.065713 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:54.065719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.067113 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.067117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:54.068159 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:54.068168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.069364 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.069367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:54.069443 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:54.069462 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 07:09:54.069517 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 07:09:54.069569 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"mutex_wait_us":21,"thread_start_us":85,"threads_started":1}
I20260924 07:09:54.069893 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:54.063752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:35469 (local address 127.0.0.1:33408)
0924 07:09:54.064484 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:54.064533 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:54.064658 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:54.065466 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:54.065469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:54.065485 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:54.065592 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:54.065599 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.067086 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.067090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:54.068274 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:54.068282 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.069225 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.069229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:54.069322 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:54.069345 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:09:54.069391 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 07:09:54.069455 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":31,"spinlock_wait_cycles":2176,"thread_start_us":87,"threads_started":1}
I20260924 07:09:54.070003 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:54.063999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33408 (local address 127.3.217.3:35469)
0924 07:09:54.065046 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:54.065056 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:54.065077 (+ 21us) server_negotiation.cc:407] Connection header received
0924 07:09:54.065153 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:54.065158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:54.065329 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:54.065715 (+ 386us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:54.065825 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.066971 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.068376 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.069107 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.069531 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:54.069701 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:54.069749 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 07:09:54.069844 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":958,"thread_start_us":119,"threads_started":1}
I20260924 07:09:54.070067 23044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:54.064549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.3.217.2:39687)
0924 07:09:54.064952 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:54.064963 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:54.065254 (+ 291us) server_negotiation.cc:407] Connection header received
0924 07:09:54.065336 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:54.065340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:54.065494 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:54.065621 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:54.065903 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.067003 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.068288 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.069034 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.069133 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:54.069310 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:54.069846 (+ 536us) server_negotiation.cc:299] Negotiation successful
0924 07:09:54.069927 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":108,"threads_started":1}
I20260924 07:09:54.072607 22831 tablet_service.cc:1467] Processing CreateTablet for tablet a753ccd78a1d4359935a900aa39ac93f (DEFAULT_TABLE table=test-table [id=14648726bf804745b5435cb15c565427]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:54.072610 22962 tablet_service.cc:1467] Processing CreateTablet for tablet a753ccd78a1d4359935a900aa39ac93f (DEFAULT_TABLE table=test-table [id=14648726bf804745b5435cb15c565427]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:54.073475 22831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a753ccd78a1d4359935a900aa39ac93f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:54.073475 22962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a753ccd78a1d4359935a900aa39ac93f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:54.076921 22700 tablet_service.cc:1467] Processing CreateTablet for tablet a753ccd78a1d4359935a900aa39ac93f (DEFAULT_TABLE table=test-table [id=14648726bf804745b5435cb15c565427]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:54.077847 22700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a753ccd78a1d4359935a900aa39ac93f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:54.079221 23045 tablet_bootstrap.cc:492] T a753ccd78a1d4359935a900aa39ac93f P 8ca41da710324f18b691e51f94e8358e: Bootstrap starting.
I20260924 07:09:54.080598 23046 tablet_bootstrap.cc:492] T a753ccd78a1d4359935a900aa39ac93f P b89905195e024ce09707c1f4a5d62271: Bootstrap starting.
I20260924 07:09:54.081863 23045 tablet_bootstrap.cc:654] T a753ccd78a1d4359935a900aa39ac93f P 8ca41da710324f18b691e51f94e8358e: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:54.082571 23045 log.cc:824] T a753ccd78a1d4359935a900aa39ac93f P 8ca41da710324f18b691e51f94e8358e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:54.082643 23046 tablet_bootstrap.cc:654] T a753ccd78a1d4359935a900aa39ac93f P b89905195e024ce09707c1f4a5d62271: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:54.083438 23046 log.cc:824] T a753ccd78a1d4359935a900aa39ac93f P b89905195e024ce09707c1f4a5d62271: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:54.084208 23045 tablet_bootstrap.cc:492] T a753ccd78a1d4359935a900aa39ac93f P 8ca41da710324f18b691e51f94e8358e: No bootstrap required, opened a new log
I20260924 07:09:54.084405 23045 ts_tablet_manager.cc:1397] T a753ccd78a1d4359935a900aa39ac93f P 8ca41da710324f18b691e51f94e8358e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:09:54.084648 23049 tablet_bootstrap.cc:492] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886: Bootstrap starting.
I20260924 07:09:54.086727 23046 tablet_bootstrap.cc:492] T a753ccd78a1d4359935a900aa39ac93f P b89905195e024ce09707c1f4a5d62271: No bootstrap required, opened a new log
I20260924 07:09:54.086911 23046 ts_tablet_manager.cc:1397] T a753ccd78a1d4359935a900aa39ac93f P b89905195e024ce09707c1f4a5d62271: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:09:54.086915 23049 tablet_bootstrap.cc:654] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:54.087694 23049 log.cc:824] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:54.087950 23045 raft_consensus.cc:348] T a753ccd78a1d4359935a900aa39ac93f P 8ca41da710324f18b691e51f94e8358e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca41da710324f18b691e51f94e8358e" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35469 } } peers { permanent_uuid: "1dd0fb0ae21c472f92ba31a119a83886" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 39429 } } peers { permanent_uuid: "b89905195e024ce09707c1f4a5d62271" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39687 } }
I20260924 07:09:54.088204 23045 raft_consensus.cc:374] T a753ccd78a1d4359935a900aa39ac93f P 8ca41da710324f18b691e51f94e8358e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:54.088286 23045 raft_consensus.cc:729] T a753ccd78a1d4359935a900aa39ac93f P 8ca41da710324f18b691e51f94e8358e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ca41da710324f18b691e51f94e8358e, State: Initialized, Role: FOLLOWER
I20260924 07:09:54.088697 23045 consensus_queue.cc:260] T a753ccd78a1d4359935a900aa39ac93f P 8ca41da710324f18b691e51f94e8358e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca41da710324f18b691e51f94e8358e" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35469 } } peers { permanent_uuid: "1dd0fb0ae21c472f92ba31a119a83886" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 39429 } } peers { permanent_uuid: "b89905195e024ce09707c1f4a5d62271" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39687 } }
I20260924 07:09:54.089314 23028 heartbeater.cc:503] Master 127.3.217.62:42511 was elected leader, sending a full tablet report...
I20260924 07:09:54.089509 23049 tablet_bootstrap.cc:492] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886: No bootstrap required, opened a new log
I20260924 07:09:54.089576 23045 ts_tablet_manager.cc:1428] T a753ccd78a1d4359935a900aa39ac93f P 8ca41da710324f18b691e51f94e8358e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 07:09:54.089675 23049 ts_tablet_manager.cc:1397] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260924 07:09:54.091233 23046 raft_consensus.cc:348] T a753ccd78a1d4359935a900aa39ac93f P b89905195e024ce09707c1f4a5d62271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca41da710324f18b691e51f94e8358e" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35469 } } peers { permanent_uuid: "1dd0fb0ae21c472f92ba31a119a83886" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 39429 } } peers { permanent_uuid: "b89905195e024ce09707c1f4a5d62271" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39687 } }
I20260924 07:09:54.091482 23046 raft_consensus.cc:374] T a753ccd78a1d4359935a900aa39ac93f P b89905195e024ce09707c1f4a5d62271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:54.091558 23046 raft_consensus.cc:729] T a753ccd78a1d4359935a900aa39ac93f P b89905195e024ce09707c1f4a5d62271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89905195e024ce09707c1f4a5d62271, State: Initialized, Role: FOLLOWER
I20260924 07:09:54.091988 23046 consensus_queue.cc:260] T a753ccd78a1d4359935a900aa39ac93f P b89905195e024ce09707c1f4a5d62271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca41da710324f18b691e51f94e8358e" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35469 } } peers { permanent_uuid: "1dd0fb0ae21c472f92ba31a119a83886" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 39429 } } peers { permanent_uuid: "b89905195e024ce09707c1f4a5d62271" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39687 } }
I20260924 07:09:54.092592 22897 heartbeater.cc:503] Master 127.3.217.62:42511 was elected leader, sending a full tablet report...
I20260924 07:09:54.092928 23046 ts_tablet_manager.cc:1428] T a753ccd78a1d4359935a900aa39ac93f P b89905195e024ce09707c1f4a5d62271: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260924 07:09:54.092859 23049 raft_consensus.cc:348] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca41da710324f18b691e51f94e8358e" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35469 } } peers { permanent_uuid: "1dd0fb0ae21c472f92ba31a119a83886" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 39429 } } peers { permanent_uuid: "b89905195e024ce09707c1f4a5d62271" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39687 } }
I20260924 07:09:54.093068 23049 raft_consensus.cc:374] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:54.093119 23049 raft_consensus.cc:729] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd0fb0ae21c472f92ba31a119a83886, State: Initialized, Role: FOLLOWER
I20260924 07:09:54.093525 23049 consensus_queue.cc:260] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca41da710324f18b691e51f94e8358e" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35469 } } peers { permanent_uuid: "1dd0fb0ae21c472f92ba31a119a83886" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 39429 } } peers { permanent_uuid: "b89905195e024ce09707c1f4a5d62271" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39687 } }
W20260924 07:09:54.093750 22898 tablet.cc:2369] T a753ccd78a1d4359935a900aa39ac93f P b89905195e024ce09707c1f4a5d62271: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:09:54.094219 23049 ts_tablet_manager.cc:1428] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886: Time spent starting tablet: real 0.004s user 0.003s sys 0.002s
I20260924 07:09:54.094517 22766 heartbeater.cc:503] Master 127.3.217.62:42511 was elected leader, sending a full tablet report...
W20260924 07:09:54.147064 22767 tablet.cc:2369] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:09:54.159042 23053 raft_consensus.cc:482] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:09:54.159251 23053 raft_consensus.cc:504] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca41da710324f18b691e51f94e8358e" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35469 } } peers { permanent_uuid: "1dd0fb0ae21c472f92ba31a119a83886" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 39429 } } peers { permanent_uuid: "b89905195e024ce09707c1f4a5d62271" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39687 } }
I20260924 07:09:54.160382 23053 leader_election.cc:288] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ca41da710324f18b691e51f94e8358e (127.3.217.3:35469), b89905195e024ce09707c1f4a5d62271 (127.3.217.2:39687)
I20260924 07:09:54.164292 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:54.160843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:48805 (local address 127.3.217.3:35469)
0924 07:09:54.160947 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:54.160951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:54.161058 (+ 107us) server_negotiation.cc:407] Connection header received
0924 07:09:54.161271 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:54.161275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:54.161319 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:54.161380 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:54.161681 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.162267 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.163320 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.163746 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.163836 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:54.163896 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:54.164111 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 07:09:54.164173 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:09:54.164295 23044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:54.160748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:32781 (local address 127.3.217.2:39687)
0924 07:09:54.160943 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:54.160947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:54.161015 (+ 68us) server_negotiation.cc:407] Connection header received
0924 07:09:54.161177 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:54.161180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:54.161226 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:54.161292 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:54.161607 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.162238 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.163313 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.163757 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.163848 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:54.163903 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:54.164108 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 07:09:54.164174 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260924 07:09:54.164315 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:54.160661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:35469 (local address 127.3.217.1:48805)
0924 07:09:54.160968 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:54.160982 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:54.161094 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:54.161411 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:54.161414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:54.161430 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:54.161558 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:54.161565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.162367 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.162370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:54.163198 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:54.163207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.163849 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.163852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:54.164000 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:54.164018 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 07:09:54.164081 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 07:09:54.164162 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":67,"spinlock_wait_cycles":4992,"thread_start_us":73,"threads_started":1}
I20260924 07:09:54.164318 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:54.160640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:39687 (local address 127.3.217.1:32781)
0924 07:09:54.160926 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:54.160941 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:54.161064 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:54.161335 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:54.161338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:54.161357 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:54.161492 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:54.161498 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.162336 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.162339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:54.163197 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:54.163207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.163863 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.163866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:54.164004 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:54.164021 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 07:09:54.164082 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 07:09:54.164150 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":56,"thread_start_us":80,"threads_started":1}
I20260924 07:09:54.164976 22851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a753ccd78a1d4359935a900aa39ac93f" candidate_uuid: "1dd0fb0ae21c472f92ba31a119a83886" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89905195e024ce09707c1f4a5d62271" is_pre_election: true
I20260924 07:09:54.164976 22982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a753ccd78a1d4359935a900aa39ac93f" candidate_uuid: "1dd0fb0ae21c472f92ba31a119a83886" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ca41da710324f18b691e51f94e8358e" is_pre_election: true
I20260924 07:09:54.165225 22851 raft_consensus.cc:2454] T a753ccd78a1d4359935a900aa39ac93f P b89905195e024ce09707c1f4a5d62271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dd0fb0ae21c472f92ba31a119a83886 in term 0.
I20260924 07:09:54.165231 22982 raft_consensus.cc:2454] T a753ccd78a1d4359935a900aa39ac93f P 8ca41da710324f18b691e51f94e8358e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dd0fb0ae21c472f92ba31a119a83886 in term 0.
I20260924 07:09:54.165683 22656 leader_election.cc:302] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dd0fb0ae21c472f92ba31a119a83886, b89905195e024ce09707c1f4a5d62271; no voters:
I20260924 07:09:54.165948 23053 raft_consensus.cc:2781] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:09:54.166024 23053 raft_consensus.cc:482] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:09:54.166064 23053 raft_consensus.cc:3037] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:54.167009 23053 raft_consensus.cc:504] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca41da710324f18b691e51f94e8358e" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35469 } } peers { permanent_uuid: "1dd0fb0ae21c472f92ba31a119a83886" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 39429 } } peers { permanent_uuid: "b89905195e024ce09707c1f4a5d62271" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39687 } }
I20260924 07:09:54.167330 23053 leader_election.cc:288] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886 [CANDIDATE]: Term 1 election: Requested vote from peers 8ca41da710324f18b691e51f94e8358e (127.3.217.3:35469), b89905195e024ce09707c1f4a5d62271 (127.3.217.2:39687)
I20260924 07:09:54.167695 22982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a753ccd78a1d4359935a900aa39ac93f" candidate_uuid: "1dd0fb0ae21c472f92ba31a119a83886" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ca41da710324f18b691e51f94e8358e"
I20260924 07:09:54.167724 22851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a753ccd78a1d4359935a900aa39ac93f" candidate_uuid: "1dd0fb0ae21c472f92ba31a119a83886" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89905195e024ce09707c1f4a5d62271"
I20260924 07:09:54.167801 22982 raft_consensus.cc:3037] T a753ccd78a1d4359935a900aa39ac93f P 8ca41da710324f18b691e51f94e8358e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:54.167837 22851 raft_consensus.cc:3037] T a753ccd78a1d4359935a900aa39ac93f P b89905195e024ce09707c1f4a5d62271 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:54.168695 22982 raft_consensus.cc:2454] T a753ccd78a1d4359935a900aa39ac93f P 8ca41da710324f18b691e51f94e8358e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dd0fb0ae21c472f92ba31a119a83886 in term 1.
I20260924 07:09:54.168802 22851 raft_consensus.cc:2454] T a753ccd78a1d4359935a900aa39ac93f P b89905195e024ce09707c1f4a5d62271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dd0fb0ae21c472f92ba31a119a83886 in term 1.
I20260924 07:09:54.168991 22653 leader_election.cc:302] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dd0fb0ae21c472f92ba31a119a83886, 8ca41da710324f18b691e51f94e8358e; no voters:
I20260924 07:09:54.169441 23053 raft_consensus.cc:2781] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:54.169651 23053 raft_consensus.cc:686] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886 [term 1 LEADER]: Becoming Leader. State: Replica: 1dd0fb0ae21c472f92ba31a119a83886, State: Running, Role: LEADER
I20260924 07:09:54.169967 23053 consensus_queue.cc:237] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca41da710324f18b691e51f94e8358e" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35469 } } peers { permanent_uuid: "1dd0fb0ae21c472f92ba31a119a83886" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 39429 } } peers { permanent_uuid: "b89905195e024ce09707c1f4a5d62271" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39687 } }
I20260924 07:09:54.173406 22578 catalog_manager.cc:5495] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dd0fb0ae21c472f92ba31a119a83886 (127.3.217.1). New cstate: current_term: 1 leader_uuid: "1dd0fb0ae21c472f92ba31a119a83886" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ca41da710324f18b691e51f94e8358e" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dd0fb0ae21c472f92ba31a119a83886" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 39429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b89905195e024ce09707c1f4a5d62271" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39687 } health_report { overall_health: UNKNOWN } } }
I20260924 07:09:54.195708 23043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:54.191938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46622 (local address 127.3.217.1:39429)
0924 07:09:54.192097 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:54.192100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:54.192115 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:09:54.192181 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:54.192183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:54.192246 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:54.192341 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:09:54.192581 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.193098 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.194083 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.194306 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.194389 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:54.194762 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:09:54.195154 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:09:54.195284 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:54.195530 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 07:09:54.195571 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 07:09:54.201318 22982 raft_consensus.cc:1261] T a753ccd78a1d4359935a900aa39ac93f P 8ca41da710324f18b691e51f94e8358e [term 1 FOLLOWER]: Refusing update from remote peer 1dd0fb0ae21c472f92ba31a119a83886: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:54.201318 22851 raft_consensus.cc:1261] T a753ccd78a1d4359935a900aa39ac93f P b89905195e024ce09707c1f4a5d62271 [term 1 FOLLOWER]: Refusing update from remote peer 1dd0fb0ae21c472f92ba31a119a83886: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:54.202033 23053 consensus_queue.cc:1035] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ca41da710324f18b691e51f94e8358e" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:54.202245 23056 consensus_queue.cc:1035] T a753ccd78a1d4359935a900aa39ac93f P 1dd0fb0ae21c472f92ba31a119a83886 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89905195e024ce09707c1f4a5d62271" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 39687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:54.214382 23060 mvcc.cc:205] Tried to move back new op lower bound from 7332797621041745920 to 7332797620922560512. Current Snapshot: MvccSnapshot[applied={T|T < 7332797621041745920 or (T in {7332797621041745920})}]
I20260924 07:09:54.219213 22576 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:54.219388 22576 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:54.222297 22576 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 687b1a4b59ff4ab6b2349dcf0aaba042: Sending DeleteTablet for 3 replicas of tablet a753ccd78a1d4359935a900aa39ac93f
I20260924 07:09:54.223104 22962 tablet_service.cc:1514] Processing DeleteTablet for tablet a753ccd78a1d4359935a900aa39ac93f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:54 UTC) from {username='slave'} at 127.0.0.1:33408
I20260924 07:09:54.223168 22700 tablet_service.cc:1514] Processing DeleteTablet for tablet a753ccd78a1d4359935a900aa39ac93f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:54 UTC) from {username='slave'} at 127.0.0.1:46618
I20260924 07:09:54.223646 22831 tablet_service.cc:1514] Processing DeleteTablet for tablet a753ccd78a1d4359935a900aa39ac93f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:54 UTC) from {username='slave'} at 127.0.0.1:46960
I20260924 07:09:54.224013 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 22638
I20260924 07:09:54.224520 23071 tablet_replica.cc:330] T a753ccd78a1d4359935a900aa39ac93f P b89905195e024ce09707c1f4a5d62271: stopping tablet replica
I20260924 07:09:54.224838 23071 raft_consensus.cc:2229] T a753ccd78a1d4359935a900aa39ac93f P b89905195e024ce09707c1f4a5d62271 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:54.225183 23071 raft_consensus.cc:2258] T a753ccd78a1d4359935a900aa39ac93f P b89905195e024ce09707c1f4a5d62271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:54.226305 23069 tablet_replica.cc:330] T a753ccd78a1d4359935a900aa39ac93f P 8ca41da710324f18b691e51f94e8358e: stopping tablet replica
I20260924 07:09:54.226508 23071 ts_tablet_manager.cc:1905] T a753ccd78a1d4359935a900aa39ac93f P b89905195e024ce09707c1f4a5d62271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:54.226537 23069 raft_consensus.cc:2229] T a753ccd78a1d4359935a900aa39ac93f P 8ca41da710324f18b691e51f94e8358e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:54.226758 23069 raft_consensus.cc:2258] T a753ccd78a1d4359935a900aa39ac93f P 8ca41da710324f18b691e51f94e8358e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:54.227918 23069 ts_tablet_manager.cc:1905] T a753ccd78a1d4359935a900aa39ac93f P 8ca41da710324f18b691e51f94e8358e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:54.229059 23071 ts_tablet_manager.cc:1918] T a753ccd78a1d4359935a900aa39ac93f P b89905195e024ce09707c1f4a5d62271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:09:54.229146 23071 log.cc:1196] T a753ccd78a1d4359935a900aa39ac93f P b89905195e024ce09707c1f4a5d62271: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/a753ccd78a1d4359935a900aa39ac93f
I20260924 07:09:54.229425 23071 ts_tablet_manager.cc:1939] T a753ccd78a1d4359935a900aa39ac93f P b89905195e024ce09707c1f4a5d62271: Deleting consensus metadata
I20260924 07:09:54.229972 23069 ts_tablet_manager.cc:1918] T a753ccd78a1d4359935a900aa39ac93f P 8ca41da710324f18b691e51f94e8358e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:09:54.230051 23069 log.cc:1196] T a753ccd78a1d4359935a900aa39ac93f P 8ca41da710324f18b691e51f94e8358e: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/a753ccd78a1d4359935a900aa39ac93f
I20260924 07:09:54.230298 23069 ts_tablet_manager.cc:1939] T a753ccd78a1d4359935a900aa39ac93f P 8ca41da710324f18b691e51f94e8358e: Deleting consensus metadata
I20260924 07:09:54.231528 22565 catalog_manager.cc:4841] TS b89905195e024ce09707c1f4a5d62271 (127.3.217.2:39687): tablet a753ccd78a1d4359935a900aa39ac93f (table test-table [id=14648726bf804745b5435cb15c565427]) successfully deleted
I20260924 07:09:54.231616 22562 catalog_manager.cc:4841] TS 8ca41da710324f18b691e51f94e8358e (127.3.217.3:35469): tablet a753ccd78a1d4359935a900aa39ac93f (table test-table [id=14648726bf804745b5435cb15c565427]) successfully deleted
W20260924 07:09:54.232503 22563 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.217.1:39429 (error 108)
I20260924 07:09:54.232765 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 22769
I20260924 07:09:54.235078 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:54.234668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:39429 (local address 127.0.0.1:46630)
0924 07:09:54.234866 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:54.234966 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.1:39429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260924 07:09:54.235327 22563 catalog_manager.cc:4584] TS 1dd0fb0ae21c472f92ba31a119a83886 (127.3.217.1:39429): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a753ccd78a1d4359935a900aa39ac93f: Network error: Client connection negotiation failed: client connection to 127.3.217.1:39429: connect: Connection refused (error 111)
I20260924 07:09:54.238157 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 22900
I20260924 07:09:54.242640 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 22547
2026-09-24T07:09:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1077 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:09:54 dist-test-slave-2p7j krb5kdc[23075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:09:54 dist-test-slave-2p7j krb5kdc[23075](info): set up 2 sockets
Sep 24 07:09:54 dist-test-slave-2p7j krb5kdc[23075](info): commencing operation
krb5kdc: starting...
W20260924 07:09:55.232000 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:09:55 dist-test-slave-2p7j krb5kdc[23075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:09:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:09:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:09:55.360271 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:35627
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:35627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:55.476449 23088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:55.476692 23088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:55.476730 23088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:55.480816 23088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:09:55.480870 23088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:55.480893 23088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:55.480908 23088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:09:55.480922 23088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:09:55.486272 23088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:35627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:35627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:55.487123 23088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:55.488046 23088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:55.497052 23096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:55.497054 23093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:55.497176 23094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:55.498030 23088 server_base.cc:1034] running on GCE node
I20260924 07:09:55.498904 23088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:55.500331 23088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:55.501557 23088 hybrid_clock.cc:648] HybridClock initialized: now 1790233795501531 us; error 43 us; skew 500 ppm
Sep 24 07:09:55 dist-test-slave-2p7j krb5kdc[23075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233795, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:55.508931 23088 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:09:55.509862 23088 webserver.cc:460] Webserver started at http://127.3.217.62:34539/ using document root <none> and password file <none>
I20260924 07:09:55.510361 23088 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:55.510416 23088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:55.510571 23088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:55.512531 23088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "63dfb23c0e2f42b3ae748f2debd2c39b"
format_stamp: "Formatted at 2026-09-24 07:09:55 on dist-test-slave-2p7j"
server_key: "b836e58339506e79d49433427834237d"
server_key_iv: "ff143ed02177c431da97dd6790dc3205"
server_key_version: "encryptionkey@0"
I20260924 07:09:55.513022 23088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "63dfb23c0e2f42b3ae748f2debd2c39b"
format_stamp: "Formatted at 2026-09-24 07:09:55 on dist-test-slave-2p7j"
server_key: "b836e58339506e79d49433427834237d"
server_key_iv: "ff143ed02177c431da97dd6790dc3205"
server_key_version: "encryptionkey@0"
I20260924 07:09:55.516566 23088 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:09:55.519354 23103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:55.520579 23088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:09:55.520736 23088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "63dfb23c0e2f42b3ae748f2debd2c39b"
format_stamp: "Formatted at 2026-09-24 07:09:55 on dist-test-slave-2p7j"
server_key: "b836e58339506e79d49433427834237d"
server_key_iv: "ff143ed02177c431da97dd6790dc3205"
server_key_version: "encryptionkey@0"
I20260924 07:09:55.520960 23088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:55.542261 23088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:55.545233 23088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:55.545399 23088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:55.552141 23088 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:35627
I20260924 07:09:55.552325 23155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:35627 every 8 connection(s)
I20260924 07:09:55.553025 23088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:09:55.555653 23156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:55.556282 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 23088
I20260924 07:09:55.556417 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:09:55.556713 3940 external_mini_cluster.cc:1428] Setting key 921ccfa9137a4453febe1968521e0957
I20260924 07:09:55.561533 23156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63dfb23c0e2f42b3ae748f2debd2c39b: Bootstrap starting.
I20260924 07:09:55.564096 23156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63dfb23c0e2f42b3ae748f2debd2c39b: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:55.564965 23156 log.cc:824] T 00000000000000000000000000000000 P 63dfb23c0e2f42b3ae748f2debd2c39b: Log is configured to *not* fsync() on all Append() calls
Sep 24 07:09:55 dist-test-slave-2p7j krb5kdc[23075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:09:55.566972 23156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63dfb23c0e2f42b3ae748f2debd2c39b: No bootstrap required, opened a new log
I20260924 07:09:55.571416 23156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63dfb23c0e2f42b3ae748f2debd2c39b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63dfb23c0e2f42b3ae748f2debd2c39b" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 35627 } }
I20260924 07:09:55.571658 23156 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63dfb23c0e2f42b3ae748f2debd2c39b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:55.571714 23156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63dfb23c0e2f42b3ae748f2debd2c39b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63dfb23c0e2f42b3ae748f2debd2c39b, State: Initialized, Role: FOLLOWER
I20260924 07:09:55.571859 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:55.557964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41016 (local address 127.3.217.62:35627)
0924 07:09:55.558319 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:55.558333 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:55.558366 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:09:55.558978 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:55.558992 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:55.559287 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:55.559576 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:55.560398 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:55.561387 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:55.562531 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:55.563336 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:55.563485 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:55.566495 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:55.566500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:55.566511 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:55.566537 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:55.568540 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:55.569470 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:55.569475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:55.569479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:55.569589 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:55.569943 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:55.569946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:55.569947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:55.570311 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:55.570468 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:55.570799 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 07:09:55.571073 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":90,"threads_started":1}
I20260924 07:09:55.572150 23156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63dfb23c0e2f42b3ae748f2debd2c39b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63dfb23c0e2f42b3ae748f2debd2c39b" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 35627 } }
I20260924 07:09:55.572305 23156 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63dfb23c0e2f42b3ae748f2debd2c39b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:09:55.572366 23156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63dfb23c0e2f42b3ae748f2debd2c39b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:09:55.572453 23156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63dfb23c0e2f42b3ae748f2debd2c39b [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:55.573369 23156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63dfb23c0e2f42b3ae748f2debd2c39b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63dfb23c0e2f42b3ae748f2debd2c39b" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 35627 } }
I20260924 07:09:55.573683 23156 leader_election.cc:302] T 00000000000000000000000000000000 P 63dfb23c0e2f42b3ae748f2debd2c39b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63dfb23c0e2f42b3ae748f2debd2c39b; no voters:
I20260924 07:09:55.573985 23156 leader_election.cc:288] T 00000000000000000000000000000000 P 63dfb23c0e2f42b3ae748f2debd2c39b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:09:55.574080 23161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63dfb23c0e2f42b3ae748f2debd2c39b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:55.574273 23161 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63dfb23c0e2f42b3ae748f2debd2c39b [term 1 LEADER]: Becoming Leader. State: Replica: 63dfb23c0e2f42b3ae748f2debd2c39b, State: Running, Role: LEADER
I20260924 07:09:55.574539 23161 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63dfb23c0e2f42b3ae748f2debd2c39b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63dfb23c0e2f42b3ae748f2debd2c39b" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 35627 } }
I20260924 07:09:55.575109 23156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63dfb23c0e2f42b3ae748f2debd2c39b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:09:55.576028 23162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63dfb23c0e2f42b3ae748f2debd2c39b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63dfb23c0e2f42b3ae748f2debd2c39b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63dfb23c0e2f42b3ae748f2debd2c39b" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 35627 } } }
I20260924 07:09:55.576161 23162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63dfb23c0e2f42b3ae748f2debd2c39b [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:55.576510 23170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:09:55.576474 23163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63dfb23c0e2f42b3ae748f2debd2c39b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63dfb23c0e2f42b3ae748f2debd2c39b. Latest consensus state: current_term: 1 leader_uuid: "63dfb23c0e2f42b3ae748f2debd2c39b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63dfb23c0e2f42b3ae748f2debd2c39b" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 35627 } } }
I20260924 07:09:55.576690 23163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63dfb23c0e2f42b3ae748f2debd2c39b [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:55.579295 23170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:09:55.584086 23170 catalog_manager.cc:1347] Generated new cluster ID: 583a1822e5584f2d89cb800be911b7d1
I20260924 07:09:55.584136 23170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:09:55.596314 23170 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:09:55.597252 23170 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:09:55.602852 23170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63dfb23c0e2f42b3ae748f2debd2c39b: Generated new TSK 0
I20260924 07:09:55.603610 23170 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
Entry for principal kudu/127.3.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab.
I20260924 07:09:55.639312 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:35627
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:55.767248 23182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:55.767477 23182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:55.767545 23182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:55.771410 23182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:55.771457 23182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:55.771538 23182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:09:55.776553 23182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.1.keytab
--principal=kudu/127.3.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:35627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:55.777480 23182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:55.778376 23182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:55.787402 23187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:55.787550 23188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:55.787607 23182 server_base.cc:1034] running on GCE node
W20260924 07:09:55.787556 23190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:55.788046 23182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:55.788627 23182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:55.789808 23182 hybrid_clock.cc:648] HybridClock initialized: now 1790233795789793 us; error 30 us; skew 500 ppm
Sep 24 07:09:55 dist-test-slave-2p7j krb5kdc[23075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233795, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:55.792331 23182 init.cc:377] Logged in from keytab as kudu/127.3.217.1@KRBTEST.COM (short username kudu)
I20260924 07:09:55.793092 23182 webserver.cc:460] Webserver started at http://127.3.217.1:40673/ using document root <none> and password file <none>
I20260924 07:09:55.793596 23182 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:55.793659 23182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:55.793826 23182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:55.795699 23182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "8ea6b42f285844a8b717a84ed0a167f2"
format_stamp: "Formatted at 2026-09-24 07:09:55 on dist-test-slave-2p7j"
server_key: "c2227d33712a4c587e34f1ff002980c0"
server_key_iv: "95662ae92b0184625f28dec433b3d7ce"
server_key_version: "encryptionkey@0"
I20260924 07:09:55.796198 23182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "8ea6b42f285844a8b717a84ed0a167f2"
format_stamp: "Formatted at 2026-09-24 07:09:55 on dist-test-slave-2p7j"
server_key: "c2227d33712a4c587e34f1ff002980c0"
server_key_iv: "95662ae92b0184625f28dec433b3d7ce"
server_key_version: "encryptionkey@0"
I20260924 07:09:55.800071 23182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:09:55.802465 23197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:55.803977 23182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:09:55.804162 23182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "8ea6b42f285844a8b717a84ed0a167f2"
format_stamp: "Formatted at 2026-09-24 07:09:55 on dist-test-slave-2p7j"
server_key: "c2227d33712a4c587e34f1ff002980c0"
server_key_iv: "95662ae92b0184625f28dec433b3d7ce"
server_key_version: "encryptionkey@0"
I20260924 07:09:55.804261 23182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:55.824162 23182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:55.827010 23182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:55.827172 23182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:55.827715 23182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:55.828476 23182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:55.828522 23182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:55.828572 23182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:55.828600 23182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:55.836675 23182 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:45165
I20260924 07:09:55.836757 23310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:45165 every 8 connection(s)
I20260924 07:09:55.837510 23182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:09:55.846292 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 23182
I20260924 07:09:55.846415 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
Sep 24 07:09:55 dist-test-slave-2p7j krb5kdc[23075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233795, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.1@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:09:55.846671 3940 external_mini_cluster.cc:1428] Setting key e80857195b006672541edbd52a03aaea
I20260924 07:09:55.850199 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:55.839175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:59117 (local address 127.3.217.62:35627)
0924 07:09:55.839342 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:55.839345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:55.840070 (+ 725us) server_negotiation.cc:407] Connection header received
0924 07:09:55.841331 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:55.841333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:55.841377 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:55.841455 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:55.842490 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:55.843059 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:55.844278 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:55.844526 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:55.844614 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:55.847059 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:55.847067 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:55.847070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:55.847095 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:55.848369 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:55.848828 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:55.848831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:55.848832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:55.848899 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:55.849197 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:55.849200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:55.849202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:55.849360 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:55.849443 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:55.849899 (+ 456us) server_negotiation.cc:299] Negotiation successful
0924 07:09:55.850016 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 07:09:55.850908 23313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:55.839520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:35627 (local address 127.3.217.1:59117)
0924 07:09:55.839927 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:55.839965 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:55.841067 (+ 1102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:55.841580 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:55.841586 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:55.841916 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:55.842354 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:55.842367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:55.843186 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:55.843189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:55.844148 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:55.844158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:55.844674 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:55.844677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:55.844840 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:55.845356 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:55.845371 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:55.846856 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:55.848475 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:55.848480 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:55.848493 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:55.848736 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:55.848979 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:55.848982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:55.848984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:55.849112 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:55.849456 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:55.849462 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:55.849689 (+ 227us) client_negotiation.cc:769] Sending connection context
0924 07:09:55.849844 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 07:09:55.850132 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":103,"threads_started":1}
I20260924 07:09:55.852005 23311 heartbeater.cc:347] Connected to a master server at 127.3.217.62:35627
I20260924 07:09:55.852223 23311 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:55.852648 23311 heartbeater.cc:511] Master 127.3.217.62:35627 requested a full tablet report, sending...
I20260924 07:09:55.854123 23120 ts_manager.cc:194] Registered new tserver with Master: 8ea6b42f285844a8b717a84ed0a167f2 (127.3.217.1:45165)
WARNING: no policy specified for kudu/127.3.217.2@KRBTEST.COM; defaulting to no policy
I20260924 07:09:55.855551 23120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.1@KRBTEST.COM'} at 127.3.217.1:59117
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
Entry for principal kudu/127.3.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab.
I20260924 07:09:55.875465 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:35627
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:55.980944 23316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:55.981204 23316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:55.981248 23316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:55.985548 23316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:55.985615 23316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:55.985694 23316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:09:55.991772 23316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.2.keytab
--principal=kudu/127.3.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:35627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:55.993096 23316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:55.994374 23316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:56.003705 23324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:56.003826 23322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:56.003830 23321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:56.004035 23316 server_base.cc:1034] running on GCE node
I20260924 07:09:56.004606 23316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:56.005175 23316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:56.006343 23316 hybrid_clock.cc:648] HybridClock initialized: now 1790233796006329 us; error 27 us; skew 500 ppm
Sep 24 07:09:56 dist-test-slave-2p7j krb5kdc[23075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233796, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:56.008886 23316 init.cc:377] Logged in from keytab as kudu/127.3.217.2@KRBTEST.COM (short username kudu)
I20260924 07:09:56.009754 23316 webserver.cc:460] Webserver started at http://127.3.217.2:41795/ using document root <none> and password file <none>
I20260924 07:09:56.010349 23316 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:56.010435 23316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:56.010603 23316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:56.012571 23316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "b4159ed0b9964aacac6d6fe569ee4985"
format_stamp: "Formatted at 2026-09-24 07:09:56 on dist-test-slave-2p7j"
server_key: "c702901731144a4659f6ee004da64e01"
server_key_iv: "b88afeb4e6b52d229d0f8cc54639c142"
server_key_version: "encryptionkey@0"
I20260924 07:09:56.013084 23316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "b4159ed0b9964aacac6d6fe569ee4985"
format_stamp: "Formatted at 2026-09-24 07:09:56 on dist-test-slave-2p7j"
server_key: "c702901731144a4659f6ee004da64e01"
server_key_iv: "b88afeb4e6b52d229d0f8cc54639c142"
server_key_version: "encryptionkey@0"
I20260924 07:09:56.017076 23316 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 07:09:56.019722 23331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:56.020925 23316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:09:56.021035 23316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "b4159ed0b9964aacac6d6fe569ee4985"
format_stamp: "Formatted at 2026-09-24 07:09:56 on dist-test-slave-2p7j"
server_key: "c702901731144a4659f6ee004da64e01"
server_key_iv: "b88afeb4e6b52d229d0f8cc54639c142"
server_key_version: "encryptionkey@0"
I20260924 07:09:56.021134 23316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:56.036892 23316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:56.039688 23316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:56.039866 23316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:56.040410 23316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:56.041209 23316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:56.041261 23316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:56.041317 23316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:56.041348 23316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:56.051075 23316 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:34597
I20260924 07:09:56.051147 23444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:34597 every 8 connection(s)
I20260924 07:09:56.051963 23316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
Sep 24 07:09:56 dist-test-slave-2p7j krb5kdc[23075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233796, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.2@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:09:56.061285 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 23316
I20260924 07:09:56.061409 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:09:56.061651 3940 external_mini_cluster.cc:1428] Setting key ed28ba3d1b3e606c73dcc42a678c642b
I20260924 07:09:56.065771 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:56.053701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:59433 (local address 127.3.217.62:35627)
0924 07:09:56.053829 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:56.053833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:56.054586 (+ 753us) server_negotiation.cc:407] Connection header received
0924 07:09:56.055823 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:56.055826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:56.055879 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:56.055948 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:56.056987 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.057506 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.058586 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.058807 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.058889 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:56.061921 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:56.061925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:56.061928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:56.061957 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:56.063793 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:56.064288 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:56.064292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:56.064293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:56.064399 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:56.064745 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:56.064748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:56.064750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:56.064920 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:56.065010 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:56.065467 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 07:09:56.065589 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:09:56.066483 23447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:56.053991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:35627 (local address 127.3.217.2:59433)
0924 07:09:56.054437 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:56.054478 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:56.055627 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:56.056094 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:56.056100 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:56.056422 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:56.056854 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:56.056865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.057617 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.057620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:56.058471 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:56.058481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.058936 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.058940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:56.059094 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:56.059619 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:56.059634 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:56.061699 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:56.063900 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:56.063905 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:56.063919 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:56.064181 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:56.064490 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:56.064493 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:56.064494 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:56.064653 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:56.065020 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:56.065027 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:56.065253 (+ 226us) client_negotiation.cc:769] Sending connection context
0924 07:09:56.065411 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 07:09:56.065709 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":131,"threads_started":1}
I20260924 07:09:56.067523 23445 heartbeater.cc:347] Connected to a master server at 127.3.217.62:35627
I20260924 07:09:56.067773 23445 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:56.068328 23445 heartbeater.cc:511] Master 127.3.217.62:35627 requested a full tablet report, sending...
I20260924 07:09:56.069888 23120 ts_manager.cc:194] Registered new tserver with Master: b4159ed0b9964aacac6d6fe569ee4985 (127.3.217.2:34597)
WARNING: no policy specified for kudu/127.3.217.3@KRBTEST.COM; defaulting to no policy
I20260924 07:09:56.070642 23120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.2@KRBTEST.COM'} at 127.3.217.2:59433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
Entry for principal kudu/127.3.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab.
I20260924 07:09:56.086169 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:35627
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:56.190109 23450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:56.190353 23450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:56.190405 23450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:56.194226 23450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:56.194271 23450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:56.194324 23450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:09:56.199267 23450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.3.keytab
--principal=kudu/127.3.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:35627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:56.200311 23450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:56.201212 23450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:56.211932 23455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:56.212057 23450 server_base.cc:1034] running on GCE node
W20260924 07:09:56.212252 23458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:56.212855 23456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:56.213253 23450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:56.213815 23450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:56.214975 23450 hybrid_clock.cc:648] HybridClock initialized: now 1790233796214960 us; error 27 us; skew 500 ppm
Sep 24 07:09:56 dist-test-slave-2p7j krb5kdc[23075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233796, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:56.217521 23450 init.cc:377] Logged in from keytab as kudu/127.3.217.3@KRBTEST.COM (short username kudu)
I20260924 07:09:56.218309 23450 webserver.cc:460] Webserver started at http://127.3.217.3:42349/ using document root <none> and password file <none>
I20260924 07:09:56.218812 23450 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:56.218878 23450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:56.219058 23450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:56.221019 23450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "01bef7dcb4f748bab5349ec4642feb0a"
format_stamp: "Formatted at 2026-09-24 07:09:56 on dist-test-slave-2p7j"
server_key: "779d580e2cfac60ee92e51d9be673609"
server_key_iv: "18087c879a719569cf738109efa3423a"
server_key_version: "encryptionkey@0"
I20260924 07:09:56.221516 23450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "01bef7dcb4f748bab5349ec4642feb0a"
format_stamp: "Formatted at 2026-09-24 07:09:56 on dist-test-slave-2p7j"
server_key: "779d580e2cfac60ee92e51d9be673609"
server_key_iv: "18087c879a719569cf738109efa3423a"
server_key_version: "encryptionkey@0"
I20260924 07:09:56.224938 23450 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:09:56.227093 23465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:56.228209 23450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:09:56.228317 23450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "01bef7dcb4f748bab5349ec4642feb0a"
format_stamp: "Formatted at 2026-09-24 07:09:56 on dist-test-slave-2p7j"
server_key: "779d580e2cfac60ee92e51d9be673609"
server_key_iv: "18087c879a719569cf738109efa3423a"
server_key_version: "encryptionkey@0"
I20260924 07:09:56.228400 23450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:56.247041 23450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:56.249882 23450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:56.250047 23450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:56.250568 23450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:56.251341 23450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:56.251387 23450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:56.251437 23450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:56.251466 23450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:56.260993 23450 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:33485
I20260924 07:09:56.261030 23578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:33485 every 8 connection(s)
I20260924 07:09:56.262490 23450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:09:56.271776 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 23450
I20260924 07:09:56.271909 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:09:56.272210 3940 external_mini_cluster.cc:1428] Setting key 5db7722406d0ec24c3047bf3944d1c23
Sep 24 07:09:56 dist-test-slave-2p7j krb5kdc[23075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233796, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.3@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:09:56.278183 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:56.264559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:54805 (local address 127.3.217.62:35627)
0924 07:09:56.264802 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:56.264811 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:56.265479 (+ 668us) server_negotiation.cc:407] Connection header received
0924 07:09:56.266788 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:56.266791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:56.266841 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:56.266934 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:56.268068 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.268602 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.269739 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.269966 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.270050 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:56.273511 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:56.273519 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:56.273586 (+ 67us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:56.273634 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:56.275493 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:56.276189 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:56.276192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:56.276194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:56.276281 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:56.276604 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:56.276608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:56.276609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:56.276790 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:56.276897 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:56.277792 (+ 895us) server_negotiation.cc:299] Negotiation successful
0924 07:09:56.277992 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260924 07:09:56.278913 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:56.264872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:35627 (local address 127.3.217.3:54805)
0924 07:09:56.265336 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:56.265375 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:56.266590 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:56.267061 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:56.267067 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:56.267413 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:56.267919 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:56.267932 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.268743 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.268746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:56.269617 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:56.269627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.270116 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.270119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:56.270277 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:56.271000 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:56.271028 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:56.273256 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:56.275627 (+ 2371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:56.275632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:56.275646 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:56.276069 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:56.276358 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:56.276360 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:56.276361 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:56.276522 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:56.276980 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:56.276988 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:56.277374 (+ 386us) client_negotiation.cc:769] Sending connection context
0924 07:09:56.277685 (+ 311us) client_negotiation.cc:240] Negotiation successful
0924 07:09:56.278002 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":120,"threads_started":1}
I20260924 07:09:56.280449 23579 heartbeater.cc:347] Connected to a master server at 127.3.217.62:35627
I20260924 07:09:56.280772 23579 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:56.281278 23579 heartbeater.cc:511] Master 127.3.217.62:35627 requested a full tablet report, sending...
I20260924 07:09:56.282519 23120 ts_manager.cc:194] Registered new tserver with Master: 01bef7dcb4f748bab5349ec4642feb0a (127.3.217.3:33485)
I20260924 07:09:56.283190 23120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.217.3@KRBTEST.COM'} at 127.3.217.3:54805
I20260924 07:09:56.287273 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:09:56.297317 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:56.289335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41026 (local address 127.3.217.62:35627)
0924 07:09:56.289488 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:56.289492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:56.289647 (+ 155us) server_negotiation.cc:407] Connection header received
0924 07:09:56.289774 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:56.289777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:56.289827 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:56.289888 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:56.290552 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.291058 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.291906 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.292125 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.292208 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:56.293792 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:56.293796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:56.293799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:56.293840 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:56.295676 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:56.296079 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:56.296083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:56.296085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:56.296178 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:56.296429 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:56.296433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:56.296435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:56.296660 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:56.296789 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:56.297010 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 07:09:56.297134 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 07:09:56.300160 23120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:09:56.303086 23120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:09:56.318149 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:56.311854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:34597 (local address 127.0.0.1:54282)
0924 07:09:56.312201 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:56.312260 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:56.312382 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:56.313138 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:56.313145 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:56.313171 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:56.313318 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:56.313327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.314920 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.314923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:56.316515 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:56.316524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.317555 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.317559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:56.317721 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:56.317796 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 07:09:56.317899 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 07:09:56.318009 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":163,"thread_start_us":88,"threads_started":1}
I20260924 07:09:56.318171 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:56.312300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54282 (local address 127.3.217.2:34597)
0924 07:09:56.312625 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:56.312639 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:56.312665 (+ 26us) server_negotiation.cc:407] Connection header received
0924 07:09:56.312748 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:56.312753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:56.312949 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:56.313478 (+ 529us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:56.313631 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.314776 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.316620 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.317429 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.317543 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:56.317687 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:56.317940 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 07:09:56.318028 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":134,"threads_started":1}
I20260924 07:09:56.318166 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:56.312059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:45165 (local address 127.0.0.1:44036)
0924 07:09:56.312701 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:56.312715 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:56.312806 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:56.313817 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:56.313820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:56.313836 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:56.313969 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:56.313976 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.315203 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.315207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:56.316600 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:56.316613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.317592 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.317596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:56.317760 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:56.317795 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 07:09:56.317869 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 07:09:56.318009 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":75,"thread_start_us":106,"threads_started":1}
I20260924 07:09:56.318522 23593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:56.312543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:33485 (local address 127.0.0.1:43412)
0924 07:09:56.312744 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:56.312756 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:56.312844 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:56.313969 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:56.313972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:56.313987 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:56.314070 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:56.314076 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.315626 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.315630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:56.316786 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:56.316794 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.317576 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.317580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:56.318246 (+ 666us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:56.318269 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:09:56.318342 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 07:09:56.318407 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":561,"thread_start_us":72,"threads_started":1}
I20260924 07:09:56.318595 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:56.313053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43412 (local address 127.3.217.3:33485)
0924 07:09:56.313544 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:56.313556 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:56.313577 (+ 21us) server_negotiation.cc:407] Connection header received
0924 07:09:56.313646 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:56.313650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:56.313827 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:56.314175 (+ 348us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:56.314288 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.315294 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.316902 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.317442 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.317664 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:56.317805 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:56.318372 (+ 567us) server_negotiation.cc:299] Negotiation successful
0924 07:09:56.318461 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":195,"threads_started":1}
I20260924 07:09:56.319756 23594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:56.312996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44036 (local address 127.3.217.1:45165)
0924 07:09:56.313378 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:56.313395 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:56.313416 (+ 21us) server_negotiation.cc:407] Connection header received
0924 07:09:56.313489 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:56.313494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:56.313675 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:56.313827 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:56.314080 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.315080 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.316734 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.317471 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.319305 (+ 1834us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:56.319437 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:56.319482 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 07:09:56.319596 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":99,"threads_started":1}
I20260924 07:09:56.320904 23379 tablet_service.cc:1467] Processing CreateTablet for tablet d554e5c8f53a4fbeb186da798c5f7730 (DEFAULT_TABLE table=test-table [id=c0dcea327a7940809d48c00ed8509bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:56.320899 23513 tablet_service.cc:1467] Processing CreateTablet for tablet d554e5c8f53a4fbeb186da798c5f7730 (DEFAULT_TABLE table=test-table [id=c0dcea327a7940809d48c00ed8509bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:56.321753 23513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d554e5c8f53a4fbeb186da798c5f7730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:56.321751 23379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d554e5c8f53a4fbeb186da798c5f7730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:56.325649 23245 tablet_service.cc:1467] Processing CreateTablet for tablet d554e5c8f53a4fbeb186da798c5f7730 (DEFAULT_TABLE table=test-table [id=c0dcea327a7940809d48c00ed8509bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:56.326617 23245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d554e5c8f53a4fbeb186da798c5f7730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:56.327545 23596 tablet_bootstrap.cc:492] T d554e5c8f53a4fbeb186da798c5f7730 P 01bef7dcb4f748bab5349ec4642feb0a: Bootstrap starting.
I20260924 07:09:56.328975 23597 tablet_bootstrap.cc:492] T d554e5c8f53a4fbeb186da798c5f7730 P b4159ed0b9964aacac6d6fe569ee4985: Bootstrap starting.
I20260924 07:09:56.329900 23596 tablet_bootstrap.cc:654] T d554e5c8f53a4fbeb186da798c5f7730 P 01bef7dcb4f748bab5349ec4642feb0a: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:56.330864 23596 log.cc:824] T d554e5c8f53a4fbeb186da798c5f7730 P 01bef7dcb4f748bab5349ec4642feb0a: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:56.331346 23597 tablet_bootstrap.cc:654] T d554e5c8f53a4fbeb186da798c5f7730 P b4159ed0b9964aacac6d6fe569ee4985: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:56.332175 23597 log.cc:824] T d554e5c8f53a4fbeb186da798c5f7730 P b4159ed0b9964aacac6d6fe569ee4985: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:56.332895 23596 tablet_bootstrap.cc:492] T d554e5c8f53a4fbeb186da798c5f7730 P 01bef7dcb4f748bab5349ec4642feb0a: No bootstrap required, opened a new log
I20260924 07:09:56.333076 23596 ts_tablet_manager.cc:1397] T d554e5c8f53a4fbeb186da798c5f7730 P 01bef7dcb4f748bab5349ec4642feb0a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260924 07:09:56.334947 23597 tablet_bootstrap.cc:492] T d554e5c8f53a4fbeb186da798c5f7730 P b4159ed0b9964aacac6d6fe569ee4985: No bootstrap required, opened a new log
I20260924 07:09:56.334967 23600 tablet_bootstrap.cc:492] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2: Bootstrap starting.
I20260924 07:09:56.335220 23597 ts_tablet_manager.cc:1397] T d554e5c8f53a4fbeb186da798c5f7730 P b4159ed0b9964aacac6d6fe569ee4985: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:09:56.336866 23596 raft_consensus.cc:348] T d554e5c8f53a4fbeb186da798c5f7730 P 01bef7dcb4f748bab5349ec4642feb0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4159ed0b9964aacac6d6fe569ee4985" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 34597 } } peers { permanent_uuid: "8ea6b42f285844a8b717a84ed0a167f2" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 45165 } } peers { permanent_uuid: "01bef7dcb4f748bab5349ec4642feb0a" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 33485 } }
I20260924 07:09:56.337152 23600 tablet_bootstrap.cc:654] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:56.337150 23596 raft_consensus.cc:374] T d554e5c8f53a4fbeb186da798c5f7730 P 01bef7dcb4f748bab5349ec4642feb0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:56.337200 23596 raft_consensus.cc:729] T d554e5c8f53a4fbeb186da798c5f7730 P 01bef7dcb4f748bab5349ec4642feb0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01bef7dcb4f748bab5349ec4642feb0a, State: Initialized, Role: FOLLOWER
I20260924 07:09:56.337697 23596 consensus_queue.cc:260] T d554e5c8f53a4fbeb186da798c5f7730 P 01bef7dcb4f748bab5349ec4642feb0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4159ed0b9964aacac6d6fe569ee4985" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 34597 } } peers { permanent_uuid: "8ea6b42f285844a8b717a84ed0a167f2" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 45165 } } peers { permanent_uuid: "01bef7dcb4f748bab5349ec4642feb0a" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 33485 } }
I20260924 07:09:56.338285 23600 log.cc:824] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:56.338661 23579 heartbeater.cc:503] Master 127.3.217.62:35627 was elected leader, sending a full tablet report...
I20260924 07:09:56.338809 23596 ts_tablet_manager.cc:1428] T d554e5c8f53a4fbeb186da798c5f7730 P 01bef7dcb4f748bab5349ec4642feb0a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 07:09:56.339311 23597 raft_consensus.cc:348] T d554e5c8f53a4fbeb186da798c5f7730 P b4159ed0b9964aacac6d6fe569ee4985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4159ed0b9964aacac6d6fe569ee4985" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 34597 } } peers { permanent_uuid: "8ea6b42f285844a8b717a84ed0a167f2" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 45165 } } peers { permanent_uuid: "01bef7dcb4f748bab5349ec4642feb0a" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 33485 } }
I20260924 07:09:56.339777 23597 raft_consensus.cc:374] T d554e5c8f53a4fbeb186da798c5f7730 P b4159ed0b9964aacac6d6fe569ee4985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:56.339860 23597 raft_consensus.cc:729] T d554e5c8f53a4fbeb186da798c5f7730 P b4159ed0b9964aacac6d6fe569ee4985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4159ed0b9964aacac6d6fe569ee4985, State: Initialized, Role: FOLLOWER
I20260924 07:09:56.340240 23597 consensus_queue.cc:260] T d554e5c8f53a4fbeb186da798c5f7730 P b4159ed0b9964aacac6d6fe569ee4985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4159ed0b9964aacac6d6fe569ee4985" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 34597 } } peers { permanent_uuid: "8ea6b42f285844a8b717a84ed0a167f2" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 45165 } } peers { permanent_uuid: "01bef7dcb4f748bab5349ec4642feb0a" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 33485 } }
I20260924 07:09:56.340839 23445 heartbeater.cc:503] Master 127.3.217.62:35627 was elected leader, sending a full tablet report...
I20260924 07:09:56.341404 23597 ts_tablet_manager.cc:1428] T d554e5c8f53a4fbeb186da798c5f7730 P b4159ed0b9964aacac6d6fe569ee4985: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260924 07:09:56.342226 23600 tablet_bootstrap.cc:492] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2: No bootstrap required, opened a new log
I20260924 07:09:56.342420 23600 ts_tablet_manager.cc:1397] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 07:09:56.345628 23600 raft_consensus.cc:348] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4159ed0b9964aacac6d6fe569ee4985" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 34597 } } peers { permanent_uuid: "8ea6b42f285844a8b717a84ed0a167f2" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 45165 } } peers { permanent_uuid: "01bef7dcb4f748bab5349ec4642feb0a" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 33485 } }
I20260924 07:09:56.345819 23600 raft_consensus.cc:374] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:56.345850 23600 raft_consensus.cc:729] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea6b42f285844a8b717a84ed0a167f2, State: Initialized, Role: FOLLOWER
I20260924 07:09:56.346171 23600 consensus_queue.cc:260] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4159ed0b9964aacac6d6fe569ee4985" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 34597 } } peers { permanent_uuid: "8ea6b42f285844a8b717a84ed0a167f2" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 45165 } } peers { permanent_uuid: "01bef7dcb4f748bab5349ec4642feb0a" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 33485 } }
I20260924 07:09:56.346760 23311 heartbeater.cc:503] Master 127.3.217.62:35627 was elected leader, sending a full tablet report...
I20260924 07:09:56.346925 23600 ts_tablet_manager.cc:1428] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 07:09:56.365454 23604 raft_consensus.cc:482] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:09:56.365623 23604 raft_consensus.cc:504] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4159ed0b9964aacac6d6fe569ee4985" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 34597 } } peers { permanent_uuid: "8ea6b42f285844a8b717a84ed0a167f2" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 45165 } } peers { permanent_uuid: "01bef7dcb4f748bab5349ec4642feb0a" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 33485 } }
I20260924 07:09:56.366606 23604 leader_election.cc:288] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4159ed0b9964aacac6d6fe569ee4985 (127.3.217.2:34597), 01bef7dcb4f748bab5349ec4642feb0a (127.3.217.3:33485)
I20260924 07:09:56.370301 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:56.366913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:45341 (local address 127.3.217.2:34597)
0924 07:09:56.367048 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:56.367053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:56.367255 (+ 202us) server_negotiation.cc:407] Connection header received
0924 07:09:56.367347 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:56.367350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:56.367396 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:56.367454 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:56.367752 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.368253 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.369376 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.369771 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.369859 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:56.369915 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:56.370113 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 07:09:56.370181 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 07:09:56.370308 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:56.366928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:52967 (local address 127.3.217.3:33485)
0924 07:09:56.367048 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:56.367053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:56.367316 (+ 263us) server_negotiation.cc:407] Connection header received
0924 07:09:56.367428 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:56.367430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:56.367478 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:56.367559 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:56.367809 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.368273 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.369366 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.369762 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.369851 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:56.369910 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:56.370129 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 07:09:56.370197 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:09:56.370327 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:56.366821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:33485 (local address 127.3.217.1:52967)
0924 07:09:56.367216 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:56.367229 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:56.367332 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:56.367600 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:56.367603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:56.367618 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:56.367709 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:56.367715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.368369 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.368373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:56.369253 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:56.369262 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.369866 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.369870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:56.369970 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:56.370034 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 07:09:56.370100 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 07:09:56.370172 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":110,"thread_start_us":91,"threads_started":1}
I20260924 07:09:56.370337 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:56.366821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:34597 (local address 127.3.217.1:45341)
0924 07:09:56.367170 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:56.367181 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:56.367266 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:56.367475 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:56.367478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:56.367493 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:56.367656 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:56.367663 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.368348 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.368351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:56.369263 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:56.369272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.369883 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.369886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:56.370007 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:56.370025 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 07:09:56.370081 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 07:09:56.370152 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":75,"thread_start_us":102,"threads_started":1}
I20260924 07:09:56.371038 23533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d554e5c8f53a4fbeb186da798c5f7730" candidate_uuid: "8ea6b42f285844a8b717a84ed0a167f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01bef7dcb4f748bab5349ec4642feb0a" is_pre_election: true
I20260924 07:09:56.371089 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d554e5c8f53a4fbeb186da798c5f7730" candidate_uuid: "8ea6b42f285844a8b717a84ed0a167f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4159ed0b9964aacac6d6fe569ee4985" is_pre_election: true
I20260924 07:09:56.371325 23533 raft_consensus.cc:2454] T d554e5c8f53a4fbeb186da798c5f7730 P 01bef7dcb4f748bab5349ec4642feb0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ea6b42f285844a8b717a84ed0a167f2 in term 0.
I20260924 07:09:56.371325 23399 raft_consensus.cc:2454] T d554e5c8f53a4fbeb186da798c5f7730 P b4159ed0b9964aacac6d6fe569ee4985 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ea6b42f285844a8b717a84ed0a167f2 in term 0.
I20260924 07:09:56.371793 23198 leader_election.cc:302] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 01bef7dcb4f748bab5349ec4642feb0a, 8ea6b42f285844a8b717a84ed0a167f2, b4159ed0b9964aacac6d6fe569ee4985; no voters:
I20260924 07:09:56.372018 23604 raft_consensus.cc:2781] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:09:56.372095 23604 raft_consensus.cc:482] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:09:56.372141 23604 raft_consensus.cc:3037] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:56.373323 23604 raft_consensus.cc:504] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4159ed0b9964aacac6d6fe569ee4985" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 34597 } } peers { permanent_uuid: "8ea6b42f285844a8b717a84ed0a167f2" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 45165 } } peers { permanent_uuid: "01bef7dcb4f748bab5349ec4642feb0a" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 33485 } }
I20260924 07:09:56.373658 23604 leader_election.cc:288] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2 [CANDIDATE]: Term 1 election: Requested vote from peers b4159ed0b9964aacac6d6fe569ee4985 (127.3.217.2:34597), 01bef7dcb4f748bab5349ec4642feb0a (127.3.217.3:33485)
I20260924 07:09:56.373978 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d554e5c8f53a4fbeb186da798c5f7730" candidate_uuid: "8ea6b42f285844a8b717a84ed0a167f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4159ed0b9964aacac6d6fe569ee4985"
I20260924 07:09:56.374002 23533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d554e5c8f53a4fbeb186da798c5f7730" candidate_uuid: "8ea6b42f285844a8b717a84ed0a167f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01bef7dcb4f748bab5349ec4642feb0a"
I20260924 07:09:56.374090 23399 raft_consensus.cc:3037] T d554e5c8f53a4fbeb186da798c5f7730 P b4159ed0b9964aacac6d6fe569ee4985 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:56.374111 23533 raft_consensus.cc:3037] T d554e5c8f53a4fbeb186da798c5f7730 P 01bef7dcb4f748bab5349ec4642feb0a [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:56.375105 23533 raft_consensus.cc:2454] T d554e5c8f53a4fbeb186da798c5f7730 P 01bef7dcb4f748bab5349ec4642feb0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ea6b42f285844a8b717a84ed0a167f2 in term 1.
I20260924 07:09:56.375128 23399 raft_consensus.cc:2454] T d554e5c8f53a4fbeb186da798c5f7730 P b4159ed0b9964aacac6d6fe569ee4985 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ea6b42f285844a8b717a84ed0a167f2 in term 1.
I20260924 07:09:56.375381 23198 leader_election.cc:302] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01bef7dcb4f748bab5349ec4642feb0a, 8ea6b42f285844a8b717a84ed0a167f2; no voters:
I20260924 07:09:56.375582 23604 raft_consensus.cc:2781] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:56.375814 23604 raft_consensus.cc:686] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2 [term 1 LEADER]: Becoming Leader. State: Replica: 8ea6b42f285844a8b717a84ed0a167f2, State: Running, Role: LEADER
I20260924 07:09:56.376143 23604 consensus_queue.cc:237] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4159ed0b9964aacac6d6fe569ee4985" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 34597 } } peers { permanent_uuid: "8ea6b42f285844a8b717a84ed0a167f2" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 45165 } } peers { permanent_uuid: "01bef7dcb4f748bab5349ec4642feb0a" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 33485 } }
I20260924 07:09:56.378826 23119 catalog_manager.cc:5495] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ea6b42f285844a8b717a84ed0a167f2 (127.3.217.1). New cstate: current_term: 1 leader_uuid: "8ea6b42f285844a8b717a84ed0a167f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4159ed0b9964aacac6d6fe569ee4985" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 34597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ea6b42f285844a8b717a84ed0a167f2" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 45165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01bef7dcb4f748bab5349ec4642feb0a" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 33485 } health_report { overall_health: UNKNOWN } } }
I20260924 07:09:56.395943 23594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:56.392703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60858 (local address 127.3.217.1:45165)
0924 07:09:56.392828 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:56.392832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:56.392846 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:09:56.392931 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:56.392933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:56.392982 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:56.393073 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:09:56.393361 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.393845 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.394701 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.394918 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.394993 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:56.395259 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:09:56.395652 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:09:56.395736 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:56.395784 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 07:09:56.395826 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 07:09:56.401067 23533 raft_consensus.cc:1261] T d554e5c8f53a4fbeb186da798c5f7730 P 01bef7dcb4f748bab5349ec4642feb0a [term 1 FOLLOWER]: Refusing update from remote peer 8ea6b42f285844a8b717a84ed0a167f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:56.401067 23399 raft_consensus.cc:1261] T d554e5c8f53a4fbeb186da798c5f7730 P b4159ed0b9964aacac6d6fe569ee4985 [term 1 FOLLOWER]: Refusing update from remote peer 8ea6b42f285844a8b717a84ed0a167f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:56.401682 23604 consensus_queue.cc:1035] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01bef7dcb4f748bab5349ec4642feb0a" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 33485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:56.401863 23607 consensus_queue.cc:1035] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4159ed0b9964aacac6d6fe569ee4985" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:56.409126 23611 mvcc.cc:205] Tried to move back new op lower bound from 7332797630052540416 to 7332797629959102464. Current Snapshot: MvccSnapshot[applied={T|T < 7332797630052540416}]
I20260924 07:09:56.411247 23612 mvcc.cc:205] Tried to move back new op lower bound from 7332797630052540416 to 7332797629959102464. Current Snapshot: MvccSnapshot[applied={T|T < 7332797630052540416}]
I20260924 07:09:56.415902 23119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:56.416065 23119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:56.418373 23119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63dfb23c0e2f42b3ae748f2debd2c39b: Sending DeleteTablet for 3 replicas of tablet d554e5c8f53a4fbeb186da798c5f7730
I20260924 07:09:56.419047 23379 tablet_service.cc:1514] Processing DeleteTablet for tablet d554e5c8f53a4fbeb186da798c5f7730 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:56 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:54282
I20260924 07:09:56.419112 23245 tablet_service.cc:1514] Processing DeleteTablet for tablet d554e5c8f53a4fbeb186da798c5f7730 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:56 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:44036
I20260924 07:09:56.419462 23619 tablet_replica.cc:330] T d554e5c8f53a4fbeb186da798c5f7730 P b4159ed0b9964aacac6d6fe569ee4985: stopping tablet replica
I20260924 07:09:56.419528 23620 tablet_replica.cc:330] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2: stopping tablet replica
I20260924 07:09:56.419488 23513 tablet_service.cc:1514] Processing DeleteTablet for tablet d554e5c8f53a4fbeb186da798c5f7730 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:56 UTC) from {username='kudu', principal='kudu/127.3.217.62@KRBTEST.COM'} at 127.0.0.1:43412
I20260924 07:09:56.419797 23620 raft_consensus.cc:2229] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:09:56.419916 23619 raft_consensus.cc:2229] T d554e5c8f53a4fbeb186da798c5f7730 P b4159ed0b9964aacac6d6fe569ee4985 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:56.420102 23620 raft_consensus.cc:2258] T d554e5c8f53a4fbeb186da798c5f7730 P 8ea6b42f285844a8b717a84ed0a167f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:56.420506 23619 raft_consensus.cc:2258] T d554e5c8f53a4fbeb186da798c5f7730 P b4159ed0b9964aacac6d6fe569ee4985 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:56.420696 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 23182
I20260924 07:09:56.422356 23619 ts_tablet_manager.cc:1905] T d554e5c8f53a4fbeb186da798c5f7730 P b4159ed0b9964aacac6d6fe569ee4985: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:56.422724 23621 tablet_replica.cc:330] T d554e5c8f53a4fbeb186da798c5f7730 P 01bef7dcb4f748bab5349ec4642feb0a: stopping tablet replica
I20260924 07:09:56.422950 23621 raft_consensus.cc:2229] T d554e5c8f53a4fbeb186da798c5f7730 P 01bef7dcb4f748bab5349ec4642feb0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:56.423187 23621 raft_consensus.cc:2258] T d554e5c8f53a4fbeb186da798c5f7730 P 01bef7dcb4f748bab5349ec4642feb0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:56.424773 23621 ts_tablet_manager.cc:1905] T d554e5c8f53a4fbeb186da798c5f7730 P 01bef7dcb4f748bab5349ec4642feb0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:56.425019 23619 ts_tablet_manager.cc:1918] T d554e5c8f53a4fbeb186da798c5f7730 P b4159ed0b9964aacac6d6fe569ee4985: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:09:56.425103 23619 log.cc:1196] T d554e5c8f53a4fbeb186da798c5f7730 P b4159ed0b9964aacac6d6fe569ee4985: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/d554e5c8f53a4fbeb186da798c5f7730
I20260924 07:09:56.425401 23619 ts_tablet_manager.cc:1939] T d554e5c8f53a4fbeb186da798c5f7730 P b4159ed0b9964aacac6d6fe569ee4985: Deleting consensus metadata
I20260924 07:09:56.426779 23107 catalog_manager.cc:4841] TS b4159ed0b9964aacac6d6fe569ee4985 (127.3.217.2:34597): tablet d554e5c8f53a4fbeb186da798c5f7730 (table test-table [id=c0dcea327a7940809d48c00ed8509bdc]) successfully deleted
I20260924 07:09:56.427093 23621 ts_tablet_manager.cc:1918] T d554e5c8f53a4fbeb186da798c5f7730 P 01bef7dcb4f748bab5349ec4642feb0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:09:56.427183 23621 log.cc:1196] T d554e5c8f53a4fbeb186da798c5f7730 P 01bef7dcb4f748bab5349ec4642feb0a: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233541052846-3940-0/minicluster-data/ts-2/wal/wals/d554e5c8f53a4fbeb186da798c5f7730
I20260924 07:09:56.427462 23621 ts_tablet_manager.cc:1939] T d554e5c8f53a4fbeb186da798c5f7730 P 01bef7dcb4f748bab5349ec4642feb0a: Deleting consensus metadata
W20260924 07:09:56.428340 23104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.217.1:45165 (error 108)
I20260924 07:09:56.428702 23104 catalog_manager.cc:4841] TS 01bef7dcb4f748bab5349ec4642feb0a (127.3.217.3:33485): tablet d554e5c8f53a4fbeb186da798c5f7730 (table test-table [id=c0dcea327a7940809d48c00ed8509bdc]) successfully deleted
W20260924 07:09:56.429184 23590 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:09:56.428937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:45165 (local address 127.0.0.1:60864)
0924 07:09:56.429047 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:56.429100 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.1:45165: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 07:09:56.429214 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 23316
W20260924 07:09:56.429328 23104 catalog_manager.cc:4584] TS 8ea6b42f285844a8b717a84ed0a167f2 (127.3.217.1:45165): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d554e5c8f53a4fbeb186da798c5f7730: Network error: Client connection negotiation failed: client connection to 127.3.217.1:45165: connect: Connection reset by peer (error 104)
I20260924 07:09:56.434262 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 23450
I20260924 07:09:56.438593 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 23088
2026-09-24T07:09:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2194 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T07:09:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:09:56Z Disabled control of system clock
I20260924 07:09:56.467759 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:37859
--webserver_interface=127.3.217.62
--webserver_port=0
--builtin_ntp_servers=127.3.217.20:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:37859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:09:56.574261 23626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:56.574478 23626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:56.574512 23626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:56.578596 23626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:09:56.578663 23626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:56.578678 23626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:56.578698 23626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:09:56.578712 23626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:09:56.583629 23626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:37859
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:37859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:56.584560 23626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:56.585781 23626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:56.593993 23631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:56.594046 23626 server_base.cc:1034] running on GCE node
W20260924 07:09:56.593972 23632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:56.593993 23634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:56.595044 23626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:56.596138 23626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:56.597302 23626 hybrid_clock.cc:648] HybridClock initialized: now 1790233796597272 us; error 40 us; skew 500 ppm
I20260924 07:09:56.599247 23626 webserver.cc:460] Webserver started at http://127.3.217.62:44851/ using document root <none> and password file <none>
I20260924 07:09:56.599815 23626 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:56.599879 23626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:56.600230 23626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:56.602721 23626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "639bf440d577438b84bcbc76dc223861"
format_stamp: "Formatted at 2026-09-24 07:09:56 on dist-test-slave-2p7j"
server_key: "06e0c301fc0efb232884d5cfb64187e0"
server_key_iv: "88146695f4dcb07f47545eba52f5a123"
server_key_version: "encryptionkey@0"
I20260924 07:09:56.603360 23626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "639bf440d577438b84bcbc76dc223861"
format_stamp: "Formatted at 2026-09-24 07:09:56 on dist-test-slave-2p7j"
server_key: "06e0c301fc0efb232884d5cfb64187e0"
server_key_iv: "88146695f4dcb07f47545eba52f5a123"
server_key_version: "encryptionkey@0"
I20260924 07:09:56.607141 23626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 07:09:56.609643 23640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:56.610816 23626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 07:09:56.610921 23626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "639bf440d577438b84bcbc76dc223861"
format_stamp: "Formatted at 2026-09-24 07:09:56 on dist-test-slave-2p7j"
server_key: "06e0c301fc0efb232884d5cfb64187e0"
server_key_iv: "88146695f4dcb07f47545eba52f5a123"
server_key_version: "encryptionkey@0"
I20260924 07:09:56.611018 23626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:56.624997 23626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:56.625653 23626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:56.625829 23626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:56.632699 23626 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:37859
I20260924 07:09:56.632714 23692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:37859 every 8 connection(s)
I20260924 07:09:56.633702 23626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:09:56.636739 23693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:56.641781 23693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 639bf440d577438b84bcbc76dc223861: Bootstrap starting.
I20260924 07:09:56.643051 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 23626
I20260924 07:09:56.643170 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:09:56.643455 3940 external_mini_cluster.cc:1428] Setting key 2ccae92bd624d10902aeffe59c6badca
I20260924 07:09:56.644235 23693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 639bf440d577438b84bcbc76dc223861: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:56.645020 23693 log.cc:824] T 00000000000000000000000000000000 P 639bf440d577438b84bcbc76dc223861: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:56.646907 23693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 639bf440d577438b84bcbc76dc223861: No bootstrap required, opened a new log
I20260924 07:09:56.650704 23693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 639bf440d577438b84bcbc76dc223861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "639bf440d577438b84bcbc76dc223861" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37859 } }
I20260924 07:09:56.650910 23693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 639bf440d577438b84bcbc76dc223861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:56.650962 23693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 639bf440d577438b84bcbc76dc223861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 639bf440d577438b84bcbc76dc223861, State: Initialized, Role: FOLLOWER
I20260924 07:09:56.651388 23693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 639bf440d577438b84bcbc76dc223861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "639bf440d577438b84bcbc76dc223861" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37859 } }
I20260924 07:09:56.651566 23693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 639bf440d577438b84bcbc76dc223861 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:09:56.651631 23693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 639bf440d577438b84bcbc76dc223861 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:09:56.651719 23693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 639bf440d577438b84bcbc76dc223861 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:56.652292 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:56.644601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41174 (local address 127.3.217.62:37859)
0924 07:09:56.645148 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:56.645166 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:56.645204 (+ 38us) server_negotiation.cc:407] Connection header received
0924 07:09:56.645798 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:56.645812 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:56.646108 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:56.646363 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:56.646711 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.647778 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.648938 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.649408 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.649526 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:56.650331 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:56.650336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:56.650347 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:56.650718 (+ 371us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:56.650840 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:56.650868 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:56.650967 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:56.651137 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 07:09:56.651386 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":106,"threads_started":1}
I20260924 07:09:56.652704 23693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 639bf440d577438b84bcbc76dc223861 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "639bf440d577438b84bcbc76dc223861" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37859 } }
I20260924 07:09:56.653007 23693 leader_election.cc:302] T 00000000000000000000000000000000 P 639bf440d577438b84bcbc76dc223861 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 639bf440d577438b84bcbc76dc223861; no voters:
I20260924 07:09:56.653250 23693 leader_election.cc:288] T 00000000000000000000000000000000 P 639bf440d577438b84bcbc76dc223861 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:09:56.653337 23698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 639bf440d577438b84bcbc76dc223861 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:56.653483 23698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 639bf440d577438b84bcbc76dc223861 [term 1 LEADER]: Becoming Leader. State: Replica: 639bf440d577438b84bcbc76dc223861, State: Running, Role: LEADER
I20260924 07:09:56.653751 23698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 639bf440d577438b84bcbc76dc223861 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "639bf440d577438b84bcbc76dc223861" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37859 } }
I20260924 07:09:56.654387 23693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 639bf440d577438b84bcbc76dc223861 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:09:56.654815 23698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 639bf440d577438b84bcbc76dc223861 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 639bf440d577438b84bcbc76dc223861. Latest consensus state: current_term: 1 leader_uuid: "639bf440d577438b84bcbc76dc223861" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "639bf440d577438b84bcbc76dc223861" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37859 } } }
I20260924 07:09:56.654927 23698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 639bf440d577438b84bcbc76dc223861 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:56.655260 23699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 639bf440d577438b84bcbc76dc223861 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "639bf440d577438b84bcbc76dc223861" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "639bf440d577438b84bcbc76dc223861" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37859 } } }
I20260924 07:09:56.655365 23699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 639bf440d577438b84bcbc76dc223861 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:56.655710 23702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:09:56.658759 23702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:09:56.664170 23702 catalog_manager.cc:1347] Generated new cluster ID: 86d1a514f7ca472bab816003972dae76
I20260924 07:09:56.664234 23702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:09:56.677124 23702 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:09:56.677928 23702 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:09:56.683239 23702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 639bf440d577438b84bcbc76dc223861: Generated new TSK 0
I20260924 07:09:56.683835 23702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:09:56.693650 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.1:0
--local_ip_for_outbound_sockets=127.3.217.1
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:37859
--builtin_ntp_servers=127.3.217.20:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:09:56.814657 23717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:56.814917 23717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:56.814960 23717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:56.818886 23717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:56.818930 23717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:56.818985 23717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.1
I20260924 07:09:56.823467 23717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.217.1
--webserver_port=0
--tserver_master_addrs=127.3.217.62:37859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.1
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:56.824491 23717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:56.825768 23717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:56.834964 23725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:56.835237 23722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:56.835269 23723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:56.835469 23717 server_base.cc:1034] running on GCE node
I20260924 07:09:56.836118 23717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:56.837172 23717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:56.838469 23717 hybrid_clock.cc:648] HybridClock initialized: now 1790233796838423 us; error 56 us; skew 500 ppm
I20260924 07:09:56.841029 23717 webserver.cc:460] Webserver started at http://127.3.217.1:41519/ using document root <none> and password file <none>
I20260924 07:09:56.841660 23717 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:56.841722 23717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:56.841892 23717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:56.844486 23717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-0/data/instance:
uuid: "d5c0b937bcc9443f92b74fbdd689fa48"
format_stamp: "Formatted at 2026-09-24 07:09:56 on dist-test-slave-2p7j"
server_key: "2d57dfc4aba0bf61791f7ec5e411a13f"
server_key_iv: "b6ef1cafb76479dfaaceed5a5c633ac3"
server_key_version: "encryptionkey@0"
I20260924 07:09:56.845096 23717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance:
uuid: "d5c0b937bcc9443f92b74fbdd689fa48"
format_stamp: "Formatted at 2026-09-24 07:09:56 on dist-test-slave-2p7j"
server_key: "2d57dfc4aba0bf61791f7ec5e411a13f"
server_key_iv: "b6ef1cafb76479dfaaceed5a5c633ac3"
server_key_version: "encryptionkey@0"
I20260924 07:09:56.849655 23717 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:09:56.852435 23731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:56.853703 23717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:09:56.853828 23717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-0/wal
uuid: "d5c0b937bcc9443f92b74fbdd689fa48"
format_stamp: "Formatted at 2026-09-24 07:09:56 on dist-test-slave-2p7j"
server_key: "2d57dfc4aba0bf61791f7ec5e411a13f"
server_key_iv: "b6ef1cafb76479dfaaceed5a5c633ac3"
server_key_version: "encryptionkey@0"
I20260924 07:09:56.853947 23717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:56.867460 23717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:56.868120 23717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:56.868291 23717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:56.868830 23717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:56.869714 23717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:56.869767 23717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:56.869819 23717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:56.869846 23717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:56.879288 23717 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.1:40991
I20260924 07:09:56.879473 23844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.1:40991 every 8 connection(s)
I20260924 07:09:56.880195 23717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-0/data/info.pb
I20260924 07:09:56.881028 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 23717
I20260924 07:09:56.881124 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-0/wal/instance
I20260924 07:09:56.881392 3940 external_mini_cluster.cc:1428] Setting key 077df5ee818a954b533554efce3b8b15
I20260924 07:09:56.883455 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.2:0
--local_ip_for_outbound_sockets=127.3.217.2
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:37859
--builtin_ntp_servers=127.3.217.20:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 07:09:56.890125 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:56.882470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.1:32809 (local address 127.3.217.62:37859)
0924 07:09:56.882604 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:56.882609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:56.883755 (+ 1146us) server_negotiation.cc:407] Connection header received
0924 07:09:56.885183 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:56.885187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:56.885242 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:56.885321 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:56.886174 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.886709 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.887965 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.888205 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.888304 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:56.889652 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:56.889655 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:56.889657 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:56.889669 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:56.889692 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:56.889701 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:56.889764 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:56.889919 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 07:09:56.889978 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:09:56.890937 23847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:56.882858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37859 (local address 127.3.217.1:32809)
0924 07:09:56.883598 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:56.883664 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:56.884974 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:56.885481 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:56.885487 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:56.885562 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:56.886027 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:56.886040 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.886834 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.886837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:56.887839 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:56.887850 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.888349 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.888354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:56.888510 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:56.889211 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:09:56.889214 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:56.889278 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:09:56.889383 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:56.889783 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:56.889793 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:56.889799 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:09:56.889910 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 07:09:56.890192 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":134,"threads_started":1}
I20260924 07:09:56.892047 23845 heartbeater.cc:347] Connected to a master server at 127.3.217.62:37859
I20260924 07:09:56.892282 23845 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:56.892868 23845 heartbeater.cc:511] Master 127.3.217.62:37859 requested a full tablet report, sending...
I20260924 07:09:56.894287 23657 ts_manager.cc:194] Registered new tserver with Master: d5c0b937bcc9443f92b74fbdd689fa48 (127.3.217.1:40991)
I20260924 07:09:56.895352 23657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.217.1:32809
W20260924 07:09:56.992300 23848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:56.992523 23848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:56.992558 23848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:56.996424 23848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:56.996471 23848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:56.996523 23848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.2
I20260924 07:09:57.001087 23848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.217.2
--webserver_port=0
--tserver_master_addrs=127.3.217.62:37859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.2
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:57.001973 23848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:57.003185 23848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:57.012228 23853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:57.012303 23854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:57.012290 23848 server_base.cc:1034] running on GCE node
W20260924 07:09:57.012220 23856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:57.012784 23848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:57.013387 23848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:57.014566 23848 hybrid_clock.cc:648] HybridClock initialized: now 1790233797014551 us; error 24 us; skew 500 ppm
I20260924 07:09:57.016301 23848 webserver.cc:460] Webserver started at http://127.3.217.2:34585/ using document root <none> and password file <none>
I20260924 07:09:57.016810 23848 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:57.016875 23848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:57.017045 23848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:57.019033 23848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-1/data/instance:
uuid: "e2382a03a7584bb3a12b84458831773b"
format_stamp: "Formatted at 2026-09-24 07:09:57 on dist-test-slave-2p7j"
server_key: "b1de7b85be5ba4380d3b2c5fc44e08d8"
server_key_iv: "3a2ffdcab3cb66c6d09dba69241aa94e"
server_key_version: "encryptionkey@0"
I20260924 07:09:57.019603 23848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance:
uuid: "e2382a03a7584bb3a12b84458831773b"
format_stamp: "Formatted at 2026-09-24 07:09:57 on dist-test-slave-2p7j"
server_key: "b1de7b85be5ba4380d3b2c5fc44e08d8"
server_key_iv: "3a2ffdcab3cb66c6d09dba69241aa94e"
server_key_version: "encryptionkey@0"
I20260924 07:09:57.023010 23848 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:09:57.025369 23862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:57.026473 23848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:09:57.026578 23848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-1/wal
uuid: "e2382a03a7584bb3a12b84458831773b"
format_stamp: "Formatted at 2026-09-24 07:09:57 on dist-test-slave-2p7j"
server_key: "b1de7b85be5ba4380d3b2c5fc44e08d8"
server_key_iv: "3a2ffdcab3cb66c6d09dba69241aa94e"
server_key_version: "encryptionkey@0"
I20260924 07:09:57.026760 23848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:57.046627 23848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:57.047237 23848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:57.047394 23848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:57.047971 23848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:57.048834 23848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:57.048882 23848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:57.048933 23848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:57.048959 23848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:57.058126 23848 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.2:42319
I20260924 07:09:57.058197 23975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.2:42319 every 8 connection(s)
I20260924 07:09:57.058931 23848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-1/data/info.pb
I20260924 07:09:57.067634 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.060704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.2:46071 (local address 127.3.217.62:37859)
0924 07:09:57.060860 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.060864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.061746 (+ 882us) server_negotiation.cc:407] Connection header received
0924 07:09:57.063047 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.063050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.063095 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.063175 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:57.064032 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.064565 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.065727 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.065957 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.066045 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.067148 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:57.067152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:57.067155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:57.067167 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:57.067185 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:57.067193 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:57.067255 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.067434 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.067482 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 07:09:57.068418 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.061021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37859 (local address 127.3.217.2:46071)
0924 07:09:57.061610 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:57.061646 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:57.062843 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:57.063309 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:57.063316 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:57.063393 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:57.063879 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.063894 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.064688 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.064691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.065561 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.065571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.066137 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.066141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.066281 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.066899 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:09:57.066900 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:57.066938 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:09:57.066998 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:57.067309 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:57.067317 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:57.067322 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:09:57.067430 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 07:09:57.067673 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":121,"threads_started":1}
I20260924 07:09:57.069160 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 23848
I20260924 07:09:57.069288 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-1/wal/instance
I20260924 07:09:57.069603 3940 external_mini_cluster.cc:1428] Setting key 9bf451af94718e1227110675ee6422f2
I20260924 07:09:57.069864 23976 heartbeater.cc:347] Connected to a master server at 127.3.217.62:37859
I20260924 07:09:57.070101 23976 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:57.070647 23976 heartbeater.cc:511] Master 127.3.217.62:37859 requested a full tablet report, sending...
I20260924 07:09:57.071882 23657 ts_manager.cc:194] Registered new tserver with Master: e2382a03a7584bb3a12b84458831773b (127.3.217.2:42319)
I20260924 07:09:57.072057 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.217.3:0
--local_ip_for_outbound_sockets=127.3.217.3
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:37859
--builtin_ntp_servers=127.3.217.20:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 07:09:57.072453 23657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.217.2:46071
W20260924 07:09:57.176244 23979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:57.176467 23979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:57.176517 23979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:57.180365 23979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:57.180409 23979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:57.180480 23979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.217.3
I20260924 07:09:57.185626 23979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.217.3
--webserver_port=0
--tserver_master_addrs=127.3.217.62:37859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.217.3
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:57.186551 23979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:57.187812 23979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:57.196712 23987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:57.196660 23984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:57.196624 23985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:57.196877 23979 server_base.cc:1034] running on GCE node
I20260924 07:09:57.197292 23979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:57.197856 23979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:57.199028 23979 hybrid_clock.cc:648] HybridClock initialized: now 1790233797199007 us; error 33 us; skew 500 ppm
I20260924 07:09:57.201859 23979 webserver.cc:460] Webserver started at http://127.3.217.3:44947/ using document root <none> and password file <none>
I20260924 07:09:57.202423 23979 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:57.202493 23979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:57.202670 23979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:57.204694 23979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-2/data/instance:
uuid: "eee4b294ff2b4701b0bb2db59b1c236e"
format_stamp: "Formatted at 2026-09-24 07:09:57 on dist-test-slave-2p7j"
server_key: "8fd5450b4acb139f608bbfd8fa2b8215"
server_key_iv: "842b29d797bac02f252a248b0736c01e"
server_key_version: "encryptionkey@0"
I20260924 07:09:57.205224 23979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance:
uuid: "eee4b294ff2b4701b0bb2db59b1c236e"
format_stamp: "Formatted at 2026-09-24 07:09:57 on dist-test-slave-2p7j"
server_key: "8fd5450b4acb139f608bbfd8fa2b8215"
server_key_iv: "842b29d797bac02f252a248b0736c01e"
server_key_version: "encryptionkey@0"
I20260924 07:09:57.208578 23979 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 07:09:57.210739 23993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:57.211807 23979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 07:09:57.211910 23979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-2/wal
uuid: "eee4b294ff2b4701b0bb2db59b1c236e"
format_stamp: "Formatted at 2026-09-24 07:09:57 on dist-test-slave-2p7j"
server_key: "8fd5450b4acb139f608bbfd8fa2b8215"
server_key_iv: "842b29d797bac02f252a248b0736c01e"
server_key_version: "encryptionkey@0"
I20260924 07:09:57.211995 23979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:57.232949 23979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:57.233517 23979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:57.233672 23979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:57.234164 23979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:57.234982 23979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:57.235028 23979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:57.235077 23979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:57.235105 23979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:57.243739 23979 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.3:33293
I20260924 07:09:57.243763 24106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.3:33293 every 8 connection(s)
I20260924 07:09:57.244524 23979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-2/data/info.pb
I20260924 07:09:57.248526 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 23979
I20260924 07:09:57.248632 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-2/wal/instance
I20260924 07:09:57.248893 3940 external_mini_cluster.cc:1428] Setting key a5ff6f2160e139b54aa195f2d001a83f
I20260924 07:09:57.253705 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.246125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:41607 (local address 127.3.217.62:37859)
0924 07:09:57.246302 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.246307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.247227 (+ 920us) server_negotiation.cc:407] Connection header received
0924 07:09:57.248670 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.248674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.248766 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.248880 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:57.249860 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.250632 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.251862 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.252077 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.252159 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.253052 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:57.253055 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:57.253058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:57.253069 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:57.253087 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:57.253094 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:57.253152 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.253530 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.253599 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 07:09:57.254745 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.246498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:37859 (local address 127.3.217.3:41607)
0924 07:09:57.247094 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:57.247130 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:57.248310 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:57.249062 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:57.249070 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:57.249163 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:57.249689 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.249707 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.250768 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.250772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.251752 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.251762 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.252187 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.252189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.252317 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.252835 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:09:57.252838 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:57.252872 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:09:57.252927 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:57.253285 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:57.253302 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:57.253311 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 07:09:57.253506 (+ 195us) client_negotiation.cc:240] Negotiation successful
0924 07:09:57.253859 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":122,"threads_started":1}
I20260924 07:09:57.255843 24107 heartbeater.cc:347] Connected to a master server at 127.3.217.62:37859
I20260924 07:09:57.256042 24107 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:57.256525 24107 heartbeater.cc:511] Master 127.3.217.62:37859 requested a full tablet report, sending...
I20260924 07:09:57.257503 23657 ts_manager.cc:194] Registered new tserver with Master: eee4b294ff2b4701b0bb2db59b1c236e (127.3.217.3:33293)
I20260924 07:09:57.258061 23657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.217.3:41607
I20260924 07:09:57.262379 3940 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:09:57.267395 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.264197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41190 (local address 127.3.217.62:37859)
0924 07:09:57.264298 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.264302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.264455 (+ 153us) server_negotiation.cc:407] Connection header received
0924 07:09:57.264544 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.264546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.264583 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.264646 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:57.264860 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.265338 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.266130 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.266367 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.266443 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.267086 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:57.267090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:57.267091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:57.267109 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:57.267129 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:57.267136 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:57.267189 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.267258 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.267302 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 07:09:57.269789 23657 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:09:57.272656 23657 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:09:57.285447 24122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.281003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.3.217.3:33293)
0924 07:09:57.281398 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.281409 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.281590 (+ 181us) server_negotiation.cc:407] Connection header received
0924 07:09:57.281664 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.281669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.281873 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.282002 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:57.282248 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.283000 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.284294 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.284827 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.284921 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.285099 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.285215 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.285301 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":106,"threads_started":1}
I20260924 07:09:57.285439 24119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.280359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.3:33293 (local address 127.0.0.1:35082)
0924 07:09:57.281382 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:57.281395 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:57.281494 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:57.282003 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:57.282008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:57.282029 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:57.282146 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.282152 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.283104 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.283108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.284198 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.284209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.284933 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.284937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.285058 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.285094 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 07:09:57.285191 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 07:09:57.285301 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":964,"mutex_wait_us":9,"thread_start_us":81,"threads_started":1}
I20260924 07:09:57.286432 24118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.280584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:42319 (local address 127.0.0.1:43690)
0924 07:09:57.281526 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:57.281538 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:57.281615 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:57.282140 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:57.282142 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:57.282157 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:57.282225 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.282230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.283632 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.283636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.285026 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.285034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.286073 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.286076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.286160 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.286183 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:09:57.286242 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 07:09:57.286319 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":234,"thread_start_us":88,"threads_started":1}
I20260924 07:09:57.286496 24123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.281038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43690 (local address 127.3.217.2:42319)
0924 07:09:57.281465 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.281479 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.281615 (+ 136us) server_negotiation.cc:407] Connection header received
0924 07:09:57.281716 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.281721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.281906 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.282046 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:57.282314 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.283479 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.285134 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.285957 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.286063 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.286222 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.286270 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.286344 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":106,"threads_started":1}
I20260924 07:09:57.286943 24121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.280239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:40991 (local address 127.0.0.1:46408)
0924 07:09:57.280901 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:57.280946 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:57.281061 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:57.282183 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:57.282186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:57.282204 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:57.282289 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.282295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.283632 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.283636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.284914 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.284924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.286574 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.286578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.286661 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.286685 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 07:09:57.286763 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 07:09:57.286824 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":69,"thread_start_us":70,"threads_started":1}
I20260924 07:09:57.287346 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.280392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.3.217.1:40991)
0924 07:09:57.281643 (+ 1251us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.281653 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.281672 (+ 19us) server_negotiation.cc:407] Connection header received
0924 07:09:57.281732 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.281736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.281906 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.282058 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:57.282408 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.283489 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.285049 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.285957 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.286063 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.286214 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.287092 (+ 878us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.287186 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1160,"thread_start_us":87,"threads_started":1}
I20260924 07:09:57.288335 24041 tablet_service.cc:1467] Processing CreateTablet for tablet f379effb75844bacb7d29116532c68eb (DEFAULT_TABLE table=test-table [id=d3534d5fc0d547b497a14c6463ee9015]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:57.288915 23910 tablet_service.cc:1467] Processing CreateTablet for tablet f379effb75844bacb7d29116532c68eb (DEFAULT_TABLE table=test-table [id=d3534d5fc0d547b497a14c6463ee9015]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:57.289191 24041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f379effb75844bacb7d29116532c68eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:57.289642 23910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f379effb75844bacb7d29116532c68eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:57.294991 24124 tablet_bootstrap.cc:492] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e: Bootstrap starting.
I20260924 07:09:57.296422 24125 tablet_bootstrap.cc:492] T f379effb75844bacb7d29116532c68eb P e2382a03a7584bb3a12b84458831773b: Bootstrap starting.
I20260924 07:09:57.297374 24124 tablet_bootstrap.cc:654] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:57.297915 24125 tablet_bootstrap.cc:654] T f379effb75844bacb7d29116532c68eb P e2382a03a7584bb3a12b84458831773b: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:57.298189 24124 log.cc:824] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:57.297914 23779 tablet_service.cc:1467] Processing CreateTablet for tablet f379effb75844bacb7d29116532c68eb (DEFAULT_TABLE table=test-table [id=d3534d5fc0d547b497a14c6463ee9015]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:57.298830 23779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f379effb75844bacb7d29116532c68eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:57.299044 24125 log.cc:824] T f379effb75844bacb7d29116532c68eb P e2382a03a7584bb3a12b84458831773b: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:57.300590 24124 tablet_bootstrap.cc:492] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e: No bootstrap required, opened a new log
I20260924 07:09:57.300803 24124 ts_tablet_manager.cc:1397] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:09:57.301560 24125 tablet_bootstrap.cc:492] T f379effb75844bacb7d29116532c68eb P e2382a03a7584bb3a12b84458831773b: No bootstrap required, opened a new log
I20260924 07:09:57.301730 24125 ts_tablet_manager.cc:1397] T f379effb75844bacb7d29116532c68eb P e2382a03a7584bb3a12b84458831773b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 07:09:57.304682 24125 raft_consensus.cc:348] T f379effb75844bacb7d29116532c68eb P e2382a03a7584bb3a12b84458831773b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c0b937bcc9443f92b74fbdd689fa48" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 40991 } } peers { permanent_uuid: "eee4b294ff2b4701b0bb2db59b1c236e" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 33293 } } peers { permanent_uuid: "e2382a03a7584bb3a12b84458831773b" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 42319 } }
I20260924 07:09:57.304879 24125 raft_consensus.cc:374] T f379effb75844bacb7d29116532c68eb P e2382a03a7584bb3a12b84458831773b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:57.304930 24125 raft_consensus.cc:729] T f379effb75844bacb7d29116532c68eb P e2382a03a7584bb3a12b84458831773b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2382a03a7584bb3a12b84458831773b, State: Initialized, Role: FOLLOWER
I20260924 07:09:57.304759 24124 raft_consensus.cc:348] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c0b937bcc9443f92b74fbdd689fa48" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 40991 } } peers { permanent_uuid: "eee4b294ff2b4701b0bb2db59b1c236e" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 33293 } } peers { permanent_uuid: "e2382a03a7584bb3a12b84458831773b" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 42319 } }
I20260924 07:09:57.304986 24124 raft_consensus.cc:374] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:57.305042 24124 raft_consensus.cc:729] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee4b294ff2b4701b0bb2db59b1c236e, State: Initialized, Role: FOLLOWER
I20260924 07:09:57.305447 24125 consensus_queue.cc:260] T f379effb75844bacb7d29116532c68eb P e2382a03a7584bb3a12b84458831773b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c0b937bcc9443f92b74fbdd689fa48" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 40991 } } peers { permanent_uuid: "eee4b294ff2b4701b0bb2db59b1c236e" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 33293 } } peers { permanent_uuid: "e2382a03a7584bb3a12b84458831773b" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 42319 } }
I20260924 07:09:57.305474 24124 consensus_queue.cc:260] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c0b937bcc9443f92b74fbdd689fa48" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 40991 } } peers { permanent_uuid: "eee4b294ff2b4701b0bb2db59b1c236e" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 33293 } } peers { permanent_uuid: "e2382a03a7584bb3a12b84458831773b" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 42319 } }
I20260924 07:09:57.305552 24128 tablet_bootstrap.cc:492] T f379effb75844bacb7d29116532c68eb P d5c0b937bcc9443f92b74fbdd689fa48: Bootstrap starting.
I20260924 07:09:57.306368 24124 ts_tablet_manager.cc:1428] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 07:09:57.306401 23976 heartbeater.cc:503] Master 127.3.217.62:37859 was elected leader, sending a full tablet report...
I20260924 07:09:57.306442 24125 ts_tablet_manager.cc:1428] T f379effb75844bacb7d29116532c68eb P e2382a03a7584bb3a12b84458831773b: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260924 07:09:57.306648 24107 heartbeater.cc:503] Master 127.3.217.62:37859 was elected leader, sending a full tablet report...
I20260924 07:09:57.307772 24128 tablet_bootstrap.cc:654] T f379effb75844bacb7d29116532c68eb P d5c0b937bcc9443f92b74fbdd689fa48: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:57.308598 24128 log.cc:824] T f379effb75844bacb7d29116532c68eb P d5c0b937bcc9443f92b74fbdd689fa48: Log is configured to *not* fsync() on all Append() calls
W20260924 07:09:57.309942 23977 tablet.cc:2369] T f379effb75844bacb7d29116532c68eb P e2382a03a7584bb3a12b84458831773b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:09:57.311192 24128 tablet_bootstrap.cc:492] T f379effb75844bacb7d29116532c68eb P d5c0b937bcc9443f92b74fbdd689fa48: No bootstrap required, opened a new log
I20260924 07:09:57.311362 24128 ts_tablet_manager.cc:1397] T f379effb75844bacb7d29116532c68eb P d5c0b937bcc9443f92b74fbdd689fa48: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:09:57.314272 24128 raft_consensus.cc:348] T f379effb75844bacb7d29116532c68eb P d5c0b937bcc9443f92b74fbdd689fa48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c0b937bcc9443f92b74fbdd689fa48" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 40991 } } peers { permanent_uuid: "eee4b294ff2b4701b0bb2db59b1c236e" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 33293 } } peers { permanent_uuid: "e2382a03a7584bb3a12b84458831773b" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 42319 } }
I20260924 07:09:57.314479 24128 raft_consensus.cc:374] T f379effb75844bacb7d29116532c68eb P d5c0b937bcc9443f92b74fbdd689fa48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:57.314518 24128 raft_consensus.cc:729] T f379effb75844bacb7d29116532c68eb P d5c0b937bcc9443f92b74fbdd689fa48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c0b937bcc9443f92b74fbdd689fa48, State: Initialized, Role: FOLLOWER
I20260924 07:09:57.314823 24128 consensus_queue.cc:260] T f379effb75844bacb7d29116532c68eb P d5c0b937bcc9443f92b74fbdd689fa48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c0b937bcc9443f92b74fbdd689fa48" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 40991 } } peers { permanent_uuid: "eee4b294ff2b4701b0bb2db59b1c236e" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 33293 } } peers { permanent_uuid: "e2382a03a7584bb3a12b84458831773b" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 42319 } }
I20260924 07:09:57.315385 23845 heartbeater.cc:503] Master 127.3.217.62:37859 was elected leader, sending a full tablet report...
I20260924 07:09:57.315675 24128 ts_tablet_manager.cc:1428] T f379effb75844bacb7d29116532c68eb P d5c0b937bcc9443f92b74fbdd689fa48: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
W20260924 07:09:57.381263 23846 tablet.cc:2369] T f379effb75844bacb7d29116532c68eb P d5c0b937bcc9443f92b74fbdd689fa48: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:09:57.495420 24108 tablet.cc:2369] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:09:57.536405 24129 raft_consensus.cc:482] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:09:57.536623 24129 raft_consensus.cc:504] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c0b937bcc9443f92b74fbdd689fa48" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 40991 } } peers { permanent_uuid: "eee4b294ff2b4701b0bb2db59b1c236e" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 33293 } } peers { permanent_uuid: "e2382a03a7584bb3a12b84458831773b" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 42319 } }
I20260924 07:09:57.537708 24129 leader_election.cc:288] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5c0b937bcc9443f92b74fbdd689fa48 (127.3.217.1:40991), e2382a03a7584bb3a12b84458831773b (127.3.217.2:42319)
I20260924 07:09:57.542447 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.538304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:45789 (local address 127.3.217.1:40991)
0924 07:09:57.538425 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.538429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.538615 (+ 186us) server_negotiation.cc:407] Connection header received
0924 07:09:57.538689 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.538691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.538728 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.538789 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:57.539001 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.539711 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.541234 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.541816 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.541933 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.541991 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.542254 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.542321 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:09:57.542447 24123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.538194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.217.3:57971 (local address 127.3.217.2:42319)
0924 07:09:57.538324 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.538329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.538362 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:09:57.538420 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.538423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.538474 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.538541 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:57.538825 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.539632 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.541234 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.541816 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.541933 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.541991 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.542249 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.542321 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 07:09:57.542450 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.538009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.2:42319 (local address 127.3.217.3:57971)
0924 07:09:57.538147 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:57.538164 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:57.538269 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:57.538581 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:57.538584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:57.538603 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:57.538720 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.538726 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.539770 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.539773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.541104 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.541110 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.541964 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.541975 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.542105 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.542146 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 07:09:57.542219 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 07:09:57.542295 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":85}
I20260924 07:09:57.542459 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.538192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:40991 (local address 127.3.217.3:45789)
0924 07:09:57.538520 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:57.538533 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:57.538615 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:57.538808 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:57.538811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:57.538827 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:57.538912 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.538916 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.539815 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.539818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.541097 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.541108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.541964 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.541968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.542100 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.542139 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 07:09:57.542219 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 07:09:57.542288 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":58,"thread_start_us":137,"threads_started":1}
I20260924 07:09:57.543057 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f379effb75844bacb7d29116532c68eb" candidate_uuid: "eee4b294ff2b4701b0bb2db59b1c236e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2382a03a7584bb3a12b84458831773b" is_pre_election: true
I20260924 07:09:57.543331 23930 raft_consensus.cc:2454] T f379effb75844bacb7d29116532c68eb P e2382a03a7584bb3a12b84458831773b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eee4b294ff2b4701b0bb2db59b1c236e in term 0.
I20260924 07:09:57.543188 23798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f379effb75844bacb7d29116532c68eb" candidate_uuid: "eee4b294ff2b4701b0bb2db59b1c236e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c0b937bcc9443f92b74fbdd689fa48" is_pre_election: true
I20260924 07:09:57.543658 23798 raft_consensus.cc:2454] T f379effb75844bacb7d29116532c68eb P d5c0b937bcc9443f92b74fbdd689fa48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eee4b294ff2b4701b0bb2db59b1c236e in term 0.
I20260924 07:09:57.544067 23994 leader_election.cc:302] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: d5c0b937bcc9443f92b74fbdd689fa48, e2382a03a7584bb3a12b84458831773b, eee4b294ff2b4701b0bb2db59b1c236e; no voters:
I20260924 07:09:57.544452 24129 raft_consensus.cc:2781] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:09:57.544541 24129 raft_consensus.cc:482] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:09:57.544571 24129 raft_consensus.cc:3037] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:57.545585 24129 raft_consensus.cc:504] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c0b937bcc9443f92b74fbdd689fa48" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 40991 } } peers { permanent_uuid: "eee4b294ff2b4701b0bb2db59b1c236e" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 33293 } } peers { permanent_uuid: "e2382a03a7584bb3a12b84458831773b" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 42319 } }
I20260924 07:09:57.545929 24129 leader_election.cc:288] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e [CANDIDATE]: Term 1 election: Requested vote from peers d5c0b937bcc9443f92b74fbdd689fa48 (127.3.217.1:40991), e2382a03a7584bb3a12b84458831773b (127.3.217.2:42319)
I20260924 07:09:57.546484 23798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f379effb75844bacb7d29116532c68eb" candidate_uuid: "eee4b294ff2b4701b0bb2db59b1c236e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c0b937bcc9443f92b74fbdd689fa48"
I20260924 07:09:57.546612 23798 raft_consensus.cc:3037] T f379effb75844bacb7d29116532c68eb P d5c0b937bcc9443f92b74fbdd689fa48 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:57.546607 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f379effb75844bacb7d29116532c68eb" candidate_uuid: "eee4b294ff2b4701b0bb2db59b1c236e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2382a03a7584bb3a12b84458831773b"
I20260924 07:09:57.546718 23930 raft_consensus.cc:3037] T f379effb75844bacb7d29116532c68eb P e2382a03a7584bb3a12b84458831773b [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:57.547721 23930 raft_consensus.cc:2454] T f379effb75844bacb7d29116532c68eb P e2382a03a7584bb3a12b84458831773b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eee4b294ff2b4701b0bb2db59b1c236e in term 1.
I20260924 07:09:57.547982 23994 leader_election.cc:302] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e2382a03a7584bb3a12b84458831773b, eee4b294ff2b4701b0bb2db59b1c236e; no voters:
I20260924 07:09:57.548130 24129 raft_consensus.cc:2781] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:57.548311 24129 raft_consensus.cc:686] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e [term 1 LEADER]: Becoming Leader. State: Replica: eee4b294ff2b4701b0bb2db59b1c236e, State: Running, Role: LEADER
I20260924 07:09:57.548326 23798 raft_consensus.cc:2454] T f379effb75844bacb7d29116532c68eb P d5c0b937bcc9443f92b74fbdd689fa48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eee4b294ff2b4701b0bb2db59b1c236e in term 1.
I20260924 07:09:57.548652 24129 consensus_queue.cc:237] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c0b937bcc9443f92b74fbdd689fa48" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 40991 } } peers { permanent_uuid: "eee4b294ff2b4701b0bb2db59b1c236e" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 33293 } } peers { permanent_uuid: "e2382a03a7584bb3a12b84458831773b" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 42319 } }
I20260924 07:09:57.551585 23657 catalog_manager.cc:5495] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e reported cstate change: term changed from 0 to 1, leader changed from <none> to eee4b294ff2b4701b0bb2db59b1c236e (127.3.217.3). New cstate: current_term: 1 leader_uuid: "eee4b294ff2b4701b0bb2db59b1c236e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c0b937bcc9443f92b74fbdd689fa48" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 40991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eee4b294ff2b4701b0bb2db59b1c236e" member_type: VOTER last_known_addr { host: "127.3.217.3" port: 33293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2382a03a7584bb3a12b84458831773b" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 42319 } health_report { overall_health: UNKNOWN } } }
I20260924 07:09:57.576013 24122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.572839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.3.217.3:33293)
0924 07:09:57.572965 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.572970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.572984 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:09:57.573055 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.573057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.573103 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.573184 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:09:57.573398 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.573902 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.574697 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.574916 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.574997 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.575258 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:09:57.575662 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:09:57.575760 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.575826 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.575870 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:09:57.581132 23798 raft_consensus.cc:1261] T f379effb75844bacb7d29116532c68eb P d5c0b937bcc9443f92b74fbdd689fa48 [term 1 FOLLOWER]: Refusing update from remote peer eee4b294ff2b4701b0bb2db59b1c236e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:57.581132 23930 raft_consensus.cc:1261] T f379effb75844bacb7d29116532c68eb P e2382a03a7584bb3a12b84458831773b [term 1 FOLLOWER]: Refusing update from remote peer eee4b294ff2b4701b0bb2db59b1c236e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:57.581790 24134 consensus_queue.cc:1035] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2382a03a7584bb3a12b84458831773b" member_type: VOTER last_known_addr { host: "127.3.217.2" port: 42319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:57.581950 24129 consensus_queue.cc:1035] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c0b937bcc9443f92b74fbdd689fa48" member_type: VOTER last_known_addr { host: "127.3.217.1" port: 40991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:57.588056 24137 mvcc.cc:205] Tried to move back new op lower bound from 7332797634885681152 to 7332797634761871360. Current Snapshot: MvccSnapshot[applied={T|T < 7332797634885681152}]
I20260924 07:09:57.592779 24140 mvcc.cc:205] Tried to move back new op lower bound from 7332797634885681152 to 7332797634761871360. Current Snapshot: MvccSnapshot[applied={T|T < 7332797634885681152}]
I20260924 07:09:57.595223 23657 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:57.595373 23657 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:57.597574 23657 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 639bf440d577438b84bcbc76dc223861: Sending DeleteTablet for 3 replicas of tablet f379effb75844bacb7d29116532c68eb
I20260924 07:09:57.598217 23779 tablet_service.cc:1514] Processing DeleteTablet for tablet f379effb75844bacb7d29116532c68eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:57 UTC) from {username='slave'} at 127.0.0.1:46408
I20260924 07:09:57.598266 24041 tablet_service.cc:1514] Processing DeleteTablet for tablet f379effb75844bacb7d29116532c68eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:57 UTC) from {username='slave'} at 127.0.0.1:35082
I20260924 07:09:57.598644 24147 tablet_replica.cc:330] T f379effb75844bacb7d29116532c68eb P d5c0b937bcc9443f92b74fbdd689fa48: stopping tablet replica
I20260924 07:09:57.598664 23910 tablet_service.cc:1514] Processing DeleteTablet for tablet f379effb75844bacb7d29116532c68eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:57 UTC) from {username='slave'} at 127.0.0.1:43690
I20260924 07:09:57.599011 24147 raft_consensus.cc:2229] T f379effb75844bacb7d29116532c68eb P d5c0b937bcc9443f92b74fbdd689fa48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:57.599068 24149 tablet_replica.cc:330] T f379effb75844bacb7d29116532c68eb P e2382a03a7584bb3a12b84458831773b: stopping tablet replica
I20260924 07:09:57.599224 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 23717
I20260924 07:09:57.599552 24147 raft_consensus.cc:2258] T f379effb75844bacb7d29116532c68eb P d5c0b937bcc9443f92b74fbdd689fa48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:57.599308 24149 raft_consensus.cc:2229] T f379effb75844bacb7d29116532c68eb P e2382a03a7584bb3a12b84458831773b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:57.600445 24149 raft_consensus.cc:2258] T f379effb75844bacb7d29116532c68eb P e2382a03a7584bb3a12b84458831773b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:57.601751 24149 ts_tablet_manager.cc:1905] T f379effb75844bacb7d29116532c68eb P e2382a03a7584bb3a12b84458831773b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:57.604490 24149 ts_tablet_manager.cc:1918] T f379effb75844bacb7d29116532c68eb P e2382a03a7584bb3a12b84458831773b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:09:57.604585 24149 log.cc:1196] T f379effb75844bacb7d29116532c68eb P e2382a03a7584bb3a12b84458831773b: Deleting WAL directory at /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233541052846-3940-0/minicluster-data/ts-1/wal/wals/f379effb75844bacb7d29116532c68eb
I20260924 07:09:57.604887 24149 ts_tablet_manager.cc:1939] T f379effb75844bacb7d29116532c68eb P e2382a03a7584bb3a12b84458831773b: Deleting consensus metadata
I20260924 07:09:57.606015 23641 catalog_manager.cc:4841] TS e2382a03a7584bb3a12b84458831773b (127.3.217.2:42319): tablet f379effb75844bacb7d29116532c68eb (table test-table [id=d3534d5fc0d547b497a14c6463ee9015]) successfully deleted
W20260924 07:09:57.606257 23644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.217.1:40991 (error 108)
I20260924 07:09:57.606858 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 23848
I20260924 07:09:57.607154 24121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.606876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.1:40991 (local address 127.0.0.1:46424)
0924 07:09:57.607001 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:57.607065 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.1:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 07:09:57.607950 23644 catalog_manager.cc:4584] TS d5c0b937bcc9443f92b74fbdd689fa48 (127.3.217.1:40991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f379effb75844bacb7d29116532c68eb: Network error: Client connection negotiation failed: client connection to 127.3.217.1:40991: connect: Connection refused (error 111)
I20260924 07:09:57.608323 24148 tablet_replica.cc:330] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e: stopping tablet replica
I20260924 07:09:57.608577 24148 raft_consensus.cc:2229] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:09:57.608875 24148 raft_consensus.cc:2258] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:57.612455 24148 ts_tablet_manager.cc:1905] T f379effb75844bacb7d29116532c68eb P eee4b294ff2b4701b0bb2db59b1c236e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:57.613297 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 23979
I20260924 07:09:57.619045 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 23626
2026-09-24T07:09:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1181 ms)
[----------] 6 tests from EncryptionPolicyTest (10136 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:09:57 dist-test-slave-2p7j krb5kdc[24153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:09:57 dist-test-slave-2p7j krb5kdc[24153](info): set up 2 sockets
Sep 24 07:09:57 dist-test-slave-2p7j krb5kdc[24153](info): commencing operation
krb5kdc: starting...
W20260924 07:09:58.584182 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:09:58 dist-test-slave-2p7j krb5kdc[24153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:09:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:09:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:09:58.702131 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35217
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:58.807937 24166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:58.808179 24166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:58.808218 24166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:58.812201 24166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:09:58.812258 24166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:58.812276 24166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:58.812291 24166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:09:58.812305 24166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:09:58.817342 24166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35217
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:09:58.818240 24166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:58.819149 24166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:58.826987 24171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:58.827057 24166 server_base.cc:1034] running on GCE node
W20260924 07:09:58.826979 24172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:58.827266 24174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:58.827823 24166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:58.828787 24166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:58.829959 24166 hybrid_clock.cc:648] HybridClock initialized: now 1790233798829943 us; error 27 us; skew 500 ppm
Sep 24 07:09:58 dist-test-slave-2p7j krb5kdc[24153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:58.833240 24166 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:09:58.834435 24166 webserver.cc:460] Webserver started at http://127.0.0.1:39967/ using document root <none> and password file <none>
I20260924 07:09:58.835095 24166 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:58.835160 24166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:58.835361 24166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:58.837538 24166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "217bc6e9c0b44a4e9a3630b6f7e56de4"
format_stamp: "Formatted at 2026-09-24 07:09:58 on dist-test-slave-2p7j"
server_key: "946009fd4e51844945ef665d2a679641"
server_key_iv: "c10e2fdc9de27188e34c88bbf202f952"
server_key_version: "encryptionkey@0"
I20260924 07:09:58.838022 24166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "217bc6e9c0b44a4e9a3630b6f7e56de4"
format_stamp: "Formatted at 2026-09-24 07:09:58 on dist-test-slave-2p7j"
server_key: "946009fd4e51844945ef665d2a679641"
server_key_iv: "c10e2fdc9de27188e34c88bbf202f952"
server_key_version: "encryptionkey@0"
I20260924 07:09:58.842854 24166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 07:09:58.846001 24181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:58.847237 24166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260924 07:09:58.847361 24166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "217bc6e9c0b44a4e9a3630b6f7e56de4"
format_stamp: "Formatted at 2026-09-24 07:09:58 on dist-test-slave-2p7j"
server_key: "946009fd4e51844945ef665d2a679641"
server_key_iv: "c10e2fdc9de27188e34c88bbf202f952"
server_key_version: "encryptionkey@0"
I20260924 07:09:58.847474 24166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:58.862617 24166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:58.865398 24166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:58.865576 24166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:58.872411 24166 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35217
I20260924 07:09:58.872432 24233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35217 every 8 connection(s)
I20260924 07:09:58.873266 24166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:09:58.875793 24234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:58.877938 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 24166
I20260924 07:09:58.878072 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:09:58.878355 3940 external_mini_cluster.cc:1428] Setting key be4a23d7647bae636fc54c77004dbc6b
I20260924 07:09:58.882186 24234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217bc6e9c0b44a4e9a3630b6f7e56de4: Bootstrap starting.
I20260924 07:09:58.884668 24234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 217bc6e9c0b44a4e9a3630b6f7e56de4: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:58.885424 24234 log.cc:824] T 00000000000000000000000000000000 P 217bc6e9c0b44a4e9a3630b6f7e56de4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:58.887616 24234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217bc6e9c0b44a4e9a3630b6f7e56de4: No bootstrap required, opened a new log
Sep 24 07:09:58 dist-test-slave-2p7j krb5kdc[24153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 07:09:58.892694 24234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 217bc6e9c0b44a4e9a3630b6f7e56de4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217bc6e9c0b44a4e9a3630b6f7e56de4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } }
I20260924 07:09:58.892969 24234 raft_consensus.cc:374] T 00000000000000000000000000000000 P 217bc6e9c0b44a4e9a3630b6f7e56de4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:58.893057 24234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 217bc6e9c0b44a4e9a3630b6f7e56de4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 217bc6e9c0b44a4e9a3630b6f7e56de4, State: Initialized, Role: FOLLOWER
I20260924 07:09:58.893503 24234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 217bc6e9c0b44a4e9a3630b6f7e56de4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217bc6e9c0b44a4e9a3630b6f7e56de4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } }
I20260924 07:09:58.893702 24234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 217bc6e9c0b44a4e9a3630b6f7e56de4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:09:58.893770 24234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 217bc6e9c0b44a4e9a3630b6f7e56de4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:09:58.893851 24234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 217bc6e9c0b44a4e9a3630b6f7e56de4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:58.894192 24237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:58.879673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36005 (local address 127.0.0.1:35217)
0924 07:09:58.880042 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:58.880057 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:58.880097 (+ 40us) server_negotiation.cc:407] Connection header received
0924 07:09:58.880841 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:58.880856 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:58.881148 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:58.881416 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:58.882596 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.883583 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.884770 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.885257 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.885388 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:58.888224 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:58.888231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:58.888246 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:58.888280 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:58.890234 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:58.891047 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:58.891052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:58.891058 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:58.891194 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:58.891635 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:58.891641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:58.891643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:58.892435 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:58.892747 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:58.892980 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 07:09:58.893211 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":100,"threads_started":1}
I20260924 07:09:58.894855 24234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 217bc6e9c0b44a4e9a3630b6f7e56de4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217bc6e9c0b44a4e9a3630b6f7e56de4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } }
I20260924 07:09:58.895153 24234 leader_election.cc:302] T 00000000000000000000000000000000 P 217bc6e9c0b44a4e9a3630b6f7e56de4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 217bc6e9c0b44a4e9a3630b6f7e56de4; no voters:
I20260924 07:09:58.895437 24234 leader_election.cc:288] T 00000000000000000000000000000000 P 217bc6e9c0b44a4e9a3630b6f7e56de4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:09:58.895545 24239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 217bc6e9c0b44a4e9a3630b6f7e56de4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:58.895696 24239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 217bc6e9c0b44a4e9a3630b6f7e56de4 [term 1 LEADER]: Becoming Leader. State: Replica: 217bc6e9c0b44a4e9a3630b6f7e56de4, State: Running, Role: LEADER
I20260924 07:09:58.896001 24239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 217bc6e9c0b44a4e9a3630b6f7e56de4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217bc6e9c0b44a4e9a3630b6f7e56de4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } }
I20260924 07:09:58.896663 24234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 217bc6e9c0b44a4e9a3630b6f7e56de4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:09:58.897084 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:09:58.898252 24240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217bc6e9c0b44a4e9a3630b6f7e56de4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "217bc6e9c0b44a4e9a3630b6f7e56de4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217bc6e9c0b44a4e9a3630b6f7e56de4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } } }
I20260924 07:09:58.898762 24240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217bc6e9c0b44a4e9a3630b6f7e56de4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:58.899152 24240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217bc6e9c0b44a4e9a3630b6f7e56de4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 217bc6e9c0b44a4e9a3630b6f7e56de4. Latest consensus state: current_term: 1 leader_uuid: "217bc6e9c0b44a4e9a3630b6f7e56de4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217bc6e9c0b44a4e9a3630b6f7e56de4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } } }
I20260924 07:09:58.899271 24240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217bc6e9c0b44a4e9a3630b6f7e56de4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:58.899329 24254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:09:58.902351 24254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:09:58.908326 24254 catalog_manager.cc:1347] Generated new cluster ID: 10b170162c8e4d8788d982a0a66fd001
I20260924 07:09:58.908382 24254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:09:58.909421 24237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:58.900357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38207 (local address 127.0.0.1:35217)
0924 07:09:58.900469 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:58.900473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:58.900903 (+ 430us) server_negotiation.cc:407] Connection header received
0924 07:09:58.901174 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:58.901177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:58.901223 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:58.901267 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:58.902223 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.902992 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.903905 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.904154 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.904246 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:58.905846 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:58.905850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:58.905853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:58.905878 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:58.907666 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:58.908116 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:58.908119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:58.908121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:58.908198 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:58.908544 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:58.908547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:58.908549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:58.908752 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:58.908893 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:58.909181 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 07:09:58.909273 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:09:58.935990 24254 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:09:58.937037 24254 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:09:58.945250 24254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 217bc6e9c0b44a4e9a3630b6f7e56de4: Generated new TSK 0
I20260924 07:09:58.945920 24254 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:09:58.953785 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 24166
W20260924 07:09:58.960577 24151 connection.cc:535] client connection to 127.0.0.1:35217 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35217): Connection reset by peer (error 104)
2026-09-24T07:09:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:09:59 dist-test-slave-2p7j krb5kdc[24266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:09:59 dist-test-slave-2p7j krb5kdc[24266](info): set up 2 sockets
Sep 24 07:09:59 dist-test-slave-2p7j krb5kdc[24266](info): commencing operation
krb5kdc: starting...
W20260924 07:09:59.923811 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:10:00 dist-test-slave-2p7j krb5kdc[24266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:10:00.041982 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37309
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:00.155125 24279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:00.155401 24279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:00.155445 24279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:00.159466 24279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:00.159539 24279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:00.159560 24279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:00.159575 24279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:00.159593 24279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:00.165071 24279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37309
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:10:00.166109 24279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:00.167080 24279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:00.175971 24287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:00.175988 24284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:00.175976 24285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:00.176532 24279 server_base.cc:1034] running on GCE node
I20260924 07:10:00.177284 24279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:00.178392 24279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:00.179592 24279 hybrid_clock.cc:648] HybridClock initialized: now 1790233800179555 us; error 54 us; skew 500 ppm
Sep 24 07:10:00 dist-test-slave-2p7j krb5kdc[24266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:00.182301 24279 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:10:00.183264 24279 webserver.cc:460] Webserver started at http://127.0.0.1:35353/ using document root <none> and password file <none>
I20260924 07:10:00.183857 24279 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:00.184111 24279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:00.184623 24279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:00.186596 24279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "020f439677d84d9a9d64e0a17bd6650b"
format_stamp: "Formatted at 2026-09-24 07:10:00 on dist-test-slave-2p7j"
server_key: "cb80db7d65310eb345d987b65517ea1a"
server_key_iv: "fed0597981fbe1a4cd66e45bdacfd758"
server_key_version: "encryptionkey@0"
I20260924 07:10:00.187114 24279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "020f439677d84d9a9d64e0a17bd6650b"
format_stamp: "Formatted at 2026-09-24 07:10:00 on dist-test-slave-2p7j"
server_key: "cb80db7d65310eb345d987b65517ea1a"
server_key_iv: "fed0597981fbe1a4cd66e45bdacfd758"
server_key_version: "encryptionkey@0"
I20260924 07:10:00.190958 24279 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:10:00.193408 24294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:00.194947 24279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:10:00.195057 24279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "020f439677d84d9a9d64e0a17bd6650b"
format_stamp: "Formatted at 2026-09-24 07:10:00 on dist-test-slave-2p7j"
server_key: "cb80db7d65310eb345d987b65517ea1a"
server_key_iv: "fed0597981fbe1a4cd66e45bdacfd758"
server_key_version: "encryptionkey@0"
I20260924 07:10:00.195151 24279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:00.235033 24279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:00.238299 24279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:00.238473 24279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:00.245431 24279 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37309
I20260924 07:10:00.245466 24346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37309 every 8 connection(s)
I20260924 07:10:00.246649 24279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:00.248363 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 24279
I20260924 07:10:00.248503 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:10:00.248816 3940 external_mini_cluster.cc:1428] Setting key e1aaf1574f1b24996ff3ad9c7f3dc030
I20260924 07:10:00.250047 24347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:00.255745 24347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 020f439677d84d9a9d64e0a17bd6650b: Bootstrap starting.
I20260924 07:10:00.257375 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:00.250095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34827 (local address 127.0.0.1:37309)
0924 07:10:00.250516 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:00.250531 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:00.250565 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:10:00.251181 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:00.251195 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:00.251555 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:00.251820 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:00.252115 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:00.253072 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:00.254208 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:00.254644 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:00.254767 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:00.255886 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:00.255893 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:00.255906 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:00.256266 (+ 360us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:00.256317 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:00.256340 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:00.256441 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:00.256603 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 07:10:00.256727 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":116,"threads_started":1}
I20260924 07:10:00.257823 24347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 020f439677d84d9a9d64e0a17bd6650b: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:00.258466 24347 log.cc:824] T 00000000000000000000000000000000 P 020f439677d84d9a9d64e0a17bd6650b: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:00.260123 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:10:00.260423 24347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 020f439677d84d9a9d64e0a17bd6650b: No bootstrap required, opened a new log
I20260924 07:10:00.264156 24347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 020f439677d84d9a9d64e0a17bd6650b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020f439677d84d9a9d64e0a17bd6650b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37309 } }
I20260924 07:10:00.264423 24347 raft_consensus.cc:374] T 00000000000000000000000000000000 P 020f439677d84d9a9d64e0a17bd6650b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:00.264482 24347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 020f439677d84d9a9d64e0a17bd6650b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 020f439677d84d9a9d64e0a17bd6650b, State: Initialized, Role: FOLLOWER
I20260924 07:10:00.264945 24347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 020f439677d84d9a9d64e0a17bd6650b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020f439677d84d9a9d64e0a17bd6650b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37309 } }
I20260924 07:10:00.265100 24347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 020f439677d84d9a9d64e0a17bd6650b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:00.265168 24347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 020f439677d84d9a9d64e0a17bd6650b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:00.265268 24347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 020f439677d84d9a9d64e0a17bd6650b [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:00.265672 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:00.261923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33417 (local address 127.0.0.1:37309)
0924 07:10:00.262029 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:00.262033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:00.262129 (+ 96us) server_negotiation.cc:407] Connection header received
0924 07:10:00.262210 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:00.262213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:00.262255 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:00.262335 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:00.262556 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:00.263306 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:00.264151 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:00.264473 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:00.264574 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:00.265293 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:00.265297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:00.265300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:00.265311 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:00.265338 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:00.265351 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:00.265438 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:00.265525 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 07:10:00.265559 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:10:00.266251 24347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 020f439677d84d9a9d64e0a17bd6650b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020f439677d84d9a9d64e0a17bd6650b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37309 } }
I20260924 07:10:00.266566 24347 leader_election.cc:302] T 00000000000000000000000000000000 P 020f439677d84d9a9d64e0a17bd6650b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 020f439677d84d9a9d64e0a17bd6650b; no voters:
I20260924 07:10:00.266830 24347 leader_election.cc:288] T 00000000000000000000000000000000 P 020f439677d84d9a9d64e0a17bd6650b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:10:00.267073 24357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 020f439677d84d9a9d64e0a17bd6650b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:00.267263 24357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 020f439677d84d9a9d64e0a17bd6650b [term 1 LEADER]: Becoming Leader. State: Replica: 020f439677d84d9a9d64e0a17bd6650b, State: Running, Role: LEADER
I20260924 07:10:00.267616 24357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 020f439677d84d9a9d64e0a17bd6650b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020f439677d84d9a9d64e0a17bd6650b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37309 } }
I20260924 07:10:00.268157 24347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 020f439677d84d9a9d64e0a17bd6650b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:00.269080 24359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 020f439677d84d9a9d64e0a17bd6650b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 020f439677d84d9a9d64e0a17bd6650b. Latest consensus state: current_term: 1 leader_uuid: "020f439677d84d9a9d64e0a17bd6650b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020f439677d84d9a9d64e0a17bd6650b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37309 } } }
I20260924 07:10:00.269214 24359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 020f439677d84d9a9d64e0a17bd6650b [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:00.269562 24366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:00.269503 24358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 020f439677d84d9a9d64e0a17bd6650b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "020f439677d84d9a9d64e0a17bd6650b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020f439677d84d9a9d64e0a17bd6650b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37309 } } }
I20260924 07:10:00.270294 24358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 020f439677d84d9a9d64e0a17bd6650b [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:00.272660 24366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:00.277467 24366 catalog_manager.cc:1347] Generated new cluster ID: 3ecdae6a76154993a9eb52ddcc853b10
I20260924 07:10:00.277516 24366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:00.284340 24366 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:00.285159 24366 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:00.290028 24366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 020f439677d84d9a9d64e0a17bd6650b: Generated new TSK 0
I20260924 07:10:00.290570 24366 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:10:00.295998 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 24279
W20260924 07:10:00.300789 24264 connection.cc:535] client connection to 127.0.0.1:37309 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37309): Connection reset by peer (error 104)
2026-09-24T07:10:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:10:00 dist-test-slave-2p7j krb5kdc[24378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:10:00 dist-test-slave-2p7j krb5kdc[24378](info): set up 2 sockets
Sep 24 07:10:00 dist-test-slave-2p7j krb5kdc[24378](info): commencing operation
krb5kdc: starting...
W20260924 07:10:01.275974 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:10:01 dist-test-slave-2p7j krb5kdc[24378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:10:01.397392 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39447
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:01.511307 24391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:01.511600 24391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:01.511659 24391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:01.515621 24391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:01.515674 24391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:01.515702 24391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:01.515730 24391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:01.515759 24391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:01.522001 24391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39447
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:10:01.523020 24391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:01.524015 24391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:01.531886 24399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:01.531870 24397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:01.531899 24396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:01.532264 24391 server_base.cc:1034] running on GCE node
I20260924 07:10:01.532775 24391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:01.533780 24391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:01.534948 24391 hybrid_clock.cc:648] HybridClock initialized: now 1790233801534917 us; error 50 us; skew 500 ppm
Sep 24 07:10:01 dist-test-slave-2p7j krb5kdc[24378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:01.543008 24391 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:10:01.544060 24391 webserver.cc:460] Webserver started at http://127.0.0.1:37599/ using document root <none> and password file <none>
I20260924 07:10:01.544687 24391 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:01.544762 24391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:01.544951 24391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:01.546833 24391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "0997fce7d32f44e9a1c4686279c84b35"
format_stamp: "Formatted at 2026-09-24 07:10:01 on dist-test-slave-2p7j"
server_key: "ceba4719f5554d7f542c368d7bea6fc5"
server_key_iv: "f8f3a8293eccbadbace951ee0148550e"
server_key_version: "encryptionkey@0"
I20260924 07:10:01.547333 24391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "0997fce7d32f44e9a1c4686279c84b35"
format_stamp: "Formatted at 2026-09-24 07:10:01 on dist-test-slave-2p7j"
server_key: "ceba4719f5554d7f542c368d7bea6fc5"
server_key_iv: "f8f3a8293eccbadbace951ee0148550e"
server_key_version: "encryptionkey@0"
I20260924 07:10:01.550755 24391 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:10:01.553462 24406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:01.554944 24391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 07:10:01.555068 24391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "0997fce7d32f44e9a1c4686279c84b35"
format_stamp: "Formatted at 2026-09-24 07:10:01 on dist-test-slave-2p7j"
server_key: "ceba4719f5554d7f542c368d7bea6fc5"
server_key_iv: "f8f3a8293eccbadbace951ee0148550e"
server_key_version: "encryptionkey@0"
I20260924 07:10:01.555159 24391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:01.559363 24391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:01.562417 24391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:01.562575 24391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:01.570400 24391 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39447
I20260924 07:10:01.570432 24458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39447 every 8 connection(s)
I20260924 07:10:01.571393 24391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:01.574002 24459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:01.574888 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 24391
I20260924 07:10:01.575027 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:10:01.575289 3940 external_mini_cluster.cc:1428] Setting key e4906d33df7f67557e061ca751c045ef
I20260924 07:10:01.579947 24459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0997fce7d32f44e9a1c4686279c84b35: Bootstrap starting.
I20260924 07:10:01.580497 24462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:01.576841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45997 (local address 127.0.0.1:39447)
0924 07:10:01.577269 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:01.577287 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:01.577323 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:10:01.577944 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:01.577960 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:01.578217 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:01.578489 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:01.579357 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:01.579362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:01.579373 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:01.579383 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:01.579434 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:01.579452 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:01.579601 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:01.579660 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 07:10:01.579796 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":134,"threads_started":1}
I20260924 07:10:01.582146 24459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0997fce7d32f44e9a1c4686279c84b35: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:01.582959 24459 log.cc:824] T 00000000000000000000000000000000 P 0997fce7d32f44e9a1c4686279c84b35: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:01.583158 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:10:01.585222 24459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0997fce7d32f44e9a1c4686279c84b35: No bootstrap required, opened a new log
I20260924 07:10:01.587037 24462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:01.585365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33257 (local address 127.0.0.1:39447)
0924 07:10:01.585465 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:01.585469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:01.585727 (+ 258us) server_negotiation.cc:407] Connection header received
0924 07:10:01.585825 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:01.585828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:01.585860 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:01.585919 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:01.586674 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:01.586678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:01.586681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:01.586691 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:01.586714 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:01.586724 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:01.586793 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:01.586892 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 07:10:01.586928 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 07:10:01.588770 24459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0997fce7d32f44e9a1c4686279c84b35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0997fce7d32f44e9a1c4686279c84b35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39447 } }
I20260924 07:10:01.588970 24459 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0997fce7d32f44e9a1c4686279c84b35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:01.589011 24459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0997fce7d32f44e9a1c4686279c84b35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0997fce7d32f44e9a1c4686279c84b35, State: Initialized, Role: FOLLOWER
I20260924 07:10:01.589407 24459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0997fce7d32f44e9a1c4686279c84b35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0997fce7d32f44e9a1c4686279c84b35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39447 } }
I20260924 07:10:01.589525 24459 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0997fce7d32f44e9a1c4686279c84b35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:01.589582 24459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0997fce7d32f44e9a1c4686279c84b35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:01.589668 24459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0997fce7d32f44e9a1c4686279c84b35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:01.590517 24459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0997fce7d32f44e9a1c4686279c84b35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0997fce7d32f44e9a1c4686279c84b35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39447 } }
I20260924 07:10:01.590808 24459 leader_election.cc:302] T 00000000000000000000000000000000 P 0997fce7d32f44e9a1c4686279c84b35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0997fce7d32f44e9a1c4686279c84b35; no voters:
I20260924 07:10:01.591059 24459 leader_election.cc:288] T 00000000000000000000000000000000 P 0997fce7d32f44e9a1c4686279c84b35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:10:01.591194 24469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0997fce7d32f44e9a1c4686279c84b35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:01.591413 24469 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0997fce7d32f44e9a1c4686279c84b35 [term 1 LEADER]: Becoming Leader. State: Replica: 0997fce7d32f44e9a1c4686279c84b35, State: Running, Role: LEADER
I20260924 07:10:01.591758 24469 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0997fce7d32f44e9a1c4686279c84b35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0997fce7d32f44e9a1c4686279c84b35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39447 } }
I20260924 07:10:01.592105 24459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0997fce7d32f44e9a1c4686279c84b35 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:01.593298 24470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0997fce7d32f44e9a1c4686279c84b35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0997fce7d32f44e9a1c4686279c84b35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0997fce7d32f44e9a1c4686279c84b35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39447 } } }
I20260924 07:10:01.593289 24471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0997fce7d32f44e9a1c4686279c84b35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0997fce7d32f44e9a1c4686279c84b35. Latest consensus state: current_term: 1 leader_uuid: "0997fce7d32f44e9a1c4686279c84b35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0997fce7d32f44e9a1c4686279c84b35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39447 } } }
I20260924 07:10:01.593413 24470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0997fce7d32f44e9a1c4686279c84b35 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:01.593413 24471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0997fce7d32f44e9a1c4686279c84b35 [sys.catalog]: This master's current role is: LEADER
W20260924 07:10:01.596423 24484 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0997fce7d32f44e9a1c4686279c84b35: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:10:01.596501 24484 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:10:01.596624 24482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:01.597436 24482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:01.602510 24482 catalog_manager.cc:1347] Generated new cluster ID: 2412afba2eb241e5a7e5f117a0a6ef86
I20260924 07:10:01.602566 24482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:01.611050 24482 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:01.611290 24482 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:01.615708 24482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0997fce7d32f44e9a1c4686279c84b35: Generated new TSK 0
I20260924 07:10:01.616608 24482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:10:01.622961 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 24391
2026-09-24T07:10:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:10:01 dist-test-slave-2p7j krb5kdc[24490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:10:01 dist-test-slave-2p7j krb5kdc[24490](info): set up 2 sockets
Sep 24 07:10:01 dist-test-slave-2p7j krb5kdc[24490](info): commencing operation
krb5kdc: starting...
W20260924 07:10:02.612488 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:10:02 dist-test-slave-2p7j krb5kdc[24490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:10:02.755611 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40227
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:02.884737 24503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:02.885090 24503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:02.885138 24503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:02.889423 24503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:02.889504 24503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:02.889523 24503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:02.889540 24503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:02.889555 24503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:02.895318 24503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:10:02.896735 24503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:02.897935 24503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:02.907889 24509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:02.907963 24503 server_base.cc:1034] running on GCE node
W20260924 07:10:02.907889 24508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:02.907894 24511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:02.908771 24503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:02.909899 24503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:02.911078 24503 hybrid_clock.cc:648] HybridClock initialized: now 1790233802911054 us; error 45 us; skew 500 ppm
Sep 24 07:10:02 dist-test-slave-2p7j krb5kdc[24490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:02.913769 24503 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:10:02.914721 24503 webserver.cc:460] Webserver started at http://127.0.0.1:32801/ using document root <none> and password file <none>
I20260924 07:10:02.915268 24503 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:02.915339 24503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:02.915557 24503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:02.917579 24503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "20098f6022fa4f86bad76e2feb27f1c1"
format_stamp: "Formatted at 2026-09-24 07:10:02 on dist-test-slave-2p7j"
server_key: "bfe00a4500468c6feeb296e309e3f4a4"
server_key_iv: "270fb6bfe34518d6043da24088abb201"
server_key_version: "encryptionkey@0"
I20260924 07:10:02.918082 24503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "20098f6022fa4f86bad76e2feb27f1c1"
format_stamp: "Formatted at 2026-09-24 07:10:02 on dist-test-slave-2p7j"
server_key: "bfe00a4500468c6feeb296e309e3f4a4"
server_key_iv: "270fb6bfe34518d6043da24088abb201"
server_key_version: "encryptionkey@0"
I20260924 07:10:02.922259 24503 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 07:10:02.925009 24518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:02.926419 24503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 07:10:02.926546 24503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "20098f6022fa4f86bad76e2feb27f1c1"
format_stamp: "Formatted at 2026-09-24 07:10:02 on dist-test-slave-2p7j"
server_key: "bfe00a4500468c6feeb296e309e3f4a4"
server_key_iv: "270fb6bfe34518d6043da24088abb201"
server_key_version: "encryptionkey@0"
I20260924 07:10:02.926658 24503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:02.952064 24503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:02.955168 24503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:02.955340 24503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:02.963081 24503 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40227
I20260924 07:10:02.963183 24570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40227 every 8 connection(s)
I20260924 07:10:02.964087 24503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:02.967806 24571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:02.973308 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 24503
I20260924 07:10:02.973549 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:10:02.974032 3940 external_mini_cluster.cc:1428] Setting key 95ca206f2a6ca645c498bcc923c9de8e
I20260924 07:10:02.975238 24571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20098f6022fa4f86bad76e2feb27f1c1: Bootstrap starting.
I20260924 07:10:02.978602 24571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20098f6022fa4f86bad76e2feb27f1c1: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:02.979681 24571 log.cc:824] T 00000000000000000000000000000000 P 20098f6022fa4f86bad76e2feb27f1c1: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:02.982363 24571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20098f6022fa4f86bad76e2feb27f1c1: No bootstrap required, opened a new log
Sep 24 07:10:02 dist-test-slave-2p7j krb5kdc[24490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 07:10:02.987198 24571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20098f6022fa4f86bad76e2feb27f1c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20098f6022fa4f86bad76e2feb27f1c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40227 } }
I20260924 07:10:02.987598 24571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20098f6022fa4f86bad76e2feb27f1c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:02.987677 24571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20098f6022fa4f86bad76e2feb27f1c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20098f6022fa4f86bad76e2feb27f1c1, State: Initialized, Role: FOLLOWER
I20260924 07:10:02.988257 24571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20098f6022fa4f86bad76e2feb27f1c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20098f6022fa4f86bad76e2feb27f1c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40227 } }
I20260924 07:10:02.988456 24571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20098f6022fa4f86bad76e2feb27f1c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:02.988541 24571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20098f6022fa4f86bad76e2feb27f1c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:02.988642 24571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20098f6022fa4f86bad76e2feb27f1c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:02.990059 24571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20098f6022fa4f86bad76e2feb27f1c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20098f6022fa4f86bad76e2feb27f1c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40227 } }
I20260924 07:10:02.990585 24571 leader_election.cc:302] T 00000000000000000000000000000000 P 20098f6022fa4f86bad76e2feb27f1c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20098f6022fa4f86bad76e2feb27f1c1; no voters:
I20260924 07:10:02.990990 24571 leader_election.cc:288] T 00000000000000000000000000000000 P 20098f6022fa4f86bad76e2feb27f1c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:10:02.991322 24576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20098f6022fa4f86bad76e2feb27f1c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:02.991674 24576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20098f6022fa4f86bad76e2feb27f1c1 [term 1 LEADER]: Becoming Leader. State: Replica: 20098f6022fa4f86bad76e2feb27f1c1, State: Running, Role: LEADER
I20260924 07:10:02.991766 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:02.975879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57109 (local address 127.0.0.1:40227)
0924 07:10:02.976556 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:02.976583 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:02.976641 (+ 58us) server_negotiation.cc:407] Connection header received
0924 07:10:02.977768 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:02.977797 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:02.978351 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:02.978812 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:02.979720 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.981113 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.982076 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.982786 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.983037 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:02.985460 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:02.985468 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:02.985493 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:02.985536 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:02.987684 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:02.988175 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:02.988182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:02.988188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:02.988348 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:02.988871 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:02.988874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:02.988876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:02.989485 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:02.989706 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:02.990121 (+ 415us) server_negotiation.cc:299] Negotiation successful
0924 07:10:02.990508 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":208,"threads_started":1}
I20260924 07:10:02.992183 24576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20098f6022fa4f86bad76e2feb27f1c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20098f6022fa4f86bad76e2feb27f1c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40227 } }
I20260924 07:10:02.992674 24571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20098f6022fa4f86bad76e2feb27f1c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:02.994046 24578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20098f6022fa4f86bad76e2feb27f1c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20098f6022fa4f86bad76e2feb27f1c1. Latest consensus state: current_term: 1 leader_uuid: "20098f6022fa4f86bad76e2feb27f1c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20098f6022fa4f86bad76e2feb27f1c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40227 } } }
I20260924 07:10:02.994197 24578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20098f6022fa4f86bad76e2feb27f1c1 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:02.994046 24577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20098f6022fa4f86bad76e2feb27f1c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20098f6022fa4f86bad76e2feb27f1c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20098f6022fa4f86bad76e2feb27f1c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40227 } } }
I20260924 07:10:02.994441 24577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20098f6022fa4f86bad76e2feb27f1c1 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:02.995025 24590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:02.995811 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:10:02.998598 24590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:03.005800 24590 catalog_manager.cc:1347] Generated new cluster ID: b8905ab9a5a14dadad5eecea132e5c1e
I20260924 07:10:03.005940 24590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:03.008050 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:02.998229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49487 (local address 127.0.0.1:40227)
0924 07:10:02.998345 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:02.998350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:02.998626 (+ 276us) server_negotiation.cc:407] Connection header received
0924 07:10:02.998786 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:02.998790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:02.998845 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:02.998912 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:02.999695 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.000268 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.001530 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.001947 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.002105 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.004014 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.004018 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.004021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.004049 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.005993 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.006582 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.006586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.006588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.006674 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.006985 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.006989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.006991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.007227 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.007339 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.007731 (+ 392us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.007878 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 07:10:03.018392 24590 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:03.019654 24590 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:03.026912 24590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20098f6022fa4f86bad76e2feb27f1c1: Generated new TSK 0
I20260924 07:10:03.027922 24590 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:10:03.034500 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 24503
2026-09-24T07:10:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:10:03 dist-test-slave-2p7j krb5kdc[24602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:10:03 dist-test-slave-2p7j krb5kdc[24602](info): set up 2 sockets
Sep 24 07:10:03 dist-test-slave-2p7j krb5kdc[24602](info): commencing operation
krb5kdc: starting...
W20260924 07:10:04.020092 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:10:04 dist-test-slave-2p7j krb5kdc[24602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:10:04.152875 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41859
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:04.279600 24615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:04.279860 24615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:04.279906 24615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:04.285080 24615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:04.285147 24615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:04.285167 24615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:04.285182 24615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:04.285197 24615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:04.290727 24615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:10:04.291800 24615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:04.292804 24615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:04.301239 24620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:04.301303 24621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:04.301364 24623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:04.301780 24615 server_base.cc:1034] running on GCE node
I20260924 07:10:04.302294 24615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:04.303321 24615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:04.304618 24615 hybrid_clock.cc:648] HybridClock initialized: now 1790233804304586 us; error 32 us; skew 500 ppm
Sep 24 07:10:04 dist-test-slave-2p7j krb5kdc[24602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:04.307857 24615 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:10:04.308737 24615 webserver.cc:460] Webserver started at http://127.0.0.1:41537/ using document root <none> and password file <none>
I20260924 07:10:04.309245 24615 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:04.309309 24615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:04.309494 24615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:04.311403 24615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "7a7428ec9e7a4c3e940e24df3b9c0792"
format_stamp: "Formatted at 2026-09-24 07:10:04 on dist-test-slave-2p7j"
server_key: "766e6ee212be84a636240365d3eaf0e3"
server_key_iv: "59fc183c6d4f0a2a0fc3b29aeed3fd44"
server_key_version: "encryptionkey@0"
I20260924 07:10:04.311954 24615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "7a7428ec9e7a4c3e940e24df3b9c0792"
format_stamp: "Formatted at 2026-09-24 07:10:04 on dist-test-slave-2p7j"
server_key: "766e6ee212be84a636240365d3eaf0e3"
server_key_iv: "59fc183c6d4f0a2a0fc3b29aeed3fd44"
server_key_version: "encryptionkey@0"
I20260924 07:10:04.316393 24615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:10:04.319664 24630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:04.321312 24615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:10:04.321436 24615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "7a7428ec9e7a4c3e940e24df3b9c0792"
format_stamp: "Formatted at 2026-09-24 07:10:04 on dist-test-slave-2p7j"
server_key: "766e6ee212be84a636240365d3eaf0e3"
server_key_iv: "59fc183c6d4f0a2a0fc3b29aeed3fd44"
server_key_version: "encryptionkey@0"
I20260924 07:10:04.321544 24615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:04.341361 24615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:04.344480 24615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:04.344651 24615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:04.351349 24615 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41859
I20260924 07:10:04.351558 24682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41859 every 8 connection(s)
I20260924 07:10:04.352310 24615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:04.355031 24683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:04.359225 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 24615
I20260924 07:10:04.359359 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:10:04.359684 3940 external_mini_cluster.cc:1428] Setting key 5c4444c83894ae8c1c0e294ff9c0dac9
I20260924 07:10:04.360653 24683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a7428ec9e7a4c3e940e24df3b9c0792: Bootstrap starting.
I20260924 07:10:04.362846 24683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a7428ec9e7a4c3e940e24df3b9c0792: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:04.363548 24683 log.cc:824] T 00000000000000000000000000000000 P 7a7428ec9e7a4c3e940e24df3b9c0792: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:04.365221 24683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a7428ec9e7a4c3e940e24df3b9c0792: No bootstrap required, opened a new log
I20260924 07:10:04.367751 24686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:04.361050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57637 (local address 127.0.0.1:41859)
0924 07:10:04.361454 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:04.361467 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:04.361499 (+ 32us) server_negotiation.cc:407] Connection header received
0924 07:10:04.362104 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:04.362119 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:04.362418 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:04.362703 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:04.363080 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:04.364090 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:04.364930 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:04.365393 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:04.365524 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:04.366225 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:04.366230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:04.366241 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:04.366587 (+ 346us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:04.366641 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:04.366664 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:04.366746 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:04.366872 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 07:10:04.367056 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":121,"threads_started":1}
I20260924 07:10:04.368669 24683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a7428ec9e7a4c3e940e24df3b9c0792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a7428ec9e7a4c3e940e24df3b9c0792" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41859 } }
I20260924 07:10:04.368918 24683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a7428ec9e7a4c3e940e24df3b9c0792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:04.368973 24683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a7428ec9e7a4c3e940e24df3b9c0792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7428ec9e7a4c3e940e24df3b9c0792, State: Initialized, Role: FOLLOWER
I20260924 07:10:04.369401 24683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a7428ec9e7a4c3e940e24df3b9c0792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a7428ec9e7a4c3e940e24df3b9c0792" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41859 } }
I20260924 07:10:04.369552 24683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a7428ec9e7a4c3e940e24df3b9c0792 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:04.369617 24683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a7428ec9e7a4c3e940e24df3b9c0792 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:04.369710 24683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a7428ec9e7a4c3e940e24df3b9c0792 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:04.370633 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:10:04.371183 24683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a7428ec9e7a4c3e940e24df3b9c0792 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a7428ec9e7a4c3e940e24df3b9c0792" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41859 } }
I20260924 07:10:04.371582 24683 leader_election.cc:302] T 00000000000000000000000000000000 P 7a7428ec9e7a4c3e940e24df3b9c0792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a7428ec9e7a4c3e940e24df3b9c0792; no voters:
I20260924 07:10:04.371906 24683 leader_election.cc:288] T 00000000000000000000000000000000 P 7a7428ec9e7a4c3e940e24df3b9c0792 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:10:04.372008 24688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a7428ec9e7a4c3e940e24df3b9c0792 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:04.372301 24688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a7428ec9e7a4c3e940e24df3b9c0792 [term 1 LEADER]: Becoming Leader. State: Replica: 7a7428ec9e7a4c3e940e24df3b9c0792, State: Running, Role: LEADER
I20260924 07:10:04.372640 24688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a7428ec9e7a4c3e940e24df3b9c0792 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a7428ec9e7a4c3e940e24df3b9c0792" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41859 } }
I20260924 07:10:04.373272 24683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a7428ec9e7a4c3e940e24df3b9c0792 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:04.373919 24688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a7428ec9e7a4c3e940e24df3b9c0792 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a7428ec9e7a4c3e940e24df3b9c0792. Latest consensus state: current_term: 1 leader_uuid: "7a7428ec9e7a4c3e940e24df3b9c0792" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a7428ec9e7a4c3e940e24df3b9c0792" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41859 } } }
I20260924 07:10:04.374950 24688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a7428ec9e7a4c3e940e24df3b9c0792 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:04.374810 24693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a7428ec9e7a4c3e940e24df3b9c0792 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a7428ec9e7a4c3e940e24df3b9c0792" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a7428ec9e7a4c3e940e24df3b9c0792" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41859 } } }
I20260924 07:10:04.375255 24693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a7428ec9e7a4c3e940e24df3b9c0792 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:04.377249 24686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:04.372853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45893 (local address 127.0.0.1:41859)
0924 07:10:04.372947 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:04.372951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:04.373187 (+ 236us) server_negotiation.cc:407] Connection header received
0924 07:10:04.373280 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:04.373282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:04.373347 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:04.373422 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:04.373698 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:04.374430 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:04.375660 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:04.375966 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:04.376060 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:04.376854 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:04.376857 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:04.376859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:04.376871 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:04.376895 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:04.376905 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:04.376962 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:04.377083 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 07:10:04.377131 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 07:10:04.378409 24708 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a7428ec9e7a4c3e940e24df3b9c0792: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:10:04.378470 24708 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:10:04.378536 24707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:04.379235 24707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:04.384675 24707 catalog_manager.cc:1347] Generated new cluster ID: 125121800202407eb31f2c017dbda387
I20260924 07:10:04.384737 24707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:04.396512 24707 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:04.397750 24707 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:04.410665 24707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a7428ec9e7a4c3e940e24df3b9c0792: Generated new TSK 0
I20260924 07:10:04.411736 24707 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:10:04.418650 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 24615
2026-09-24T07:10:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:10:04 dist-test-slave-2p7j krb5kdc[24714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:10:04 dist-test-slave-2p7j krb5kdc[24714](info): set up 2 sockets
Sep 24 07:10:04 dist-test-slave-2p7j krb5kdc[24714](info): commencing operation
krb5kdc: starting...
W20260924 07:10:05.435994 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:10:05 dist-test-slave-2p7j krb5kdc[24714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:10:05.568677 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38089
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:05.679612 24727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:05.679864 24727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:05.679906 24727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:05.684221 24727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:05.684275 24727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:05.684293 24727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:05.684309 24727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:05.684324 24727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:05.689811 24727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:10:05.691414 24727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:05.692451 24727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:05.700650 24733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:05.700635 24732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:05.700635 24735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:05.701700 24727 server_base.cc:1034] running on GCE node
I20260924 07:10:05.702196 24727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:05.703305 24727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:05.704576 24727 hybrid_clock.cc:648] HybridClock initialized: now 1790233805704536 us; error 77 us; skew 500 ppm
Sep 24 07:10:05 dist-test-slave-2p7j krb5kdc[24714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:05.707897 24727 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:10:05.708899 24727 webserver.cc:460] Webserver started at http://127.0.0.1:45155/ using document root <none> and password file <none>
I20260924 07:10:05.709544 24727 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:05.709626 24727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:05.709816 24727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:05.711942 24727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "28227c9bea4642e6b0710f5a2e737aa6"
format_stamp: "Formatted at 2026-09-24 07:10:05 on dist-test-slave-2p7j"
server_key: "f19546d5d507b862b86179d8f1393eb5"
server_key_iv: "bf44ec318de40ff72e462946eb21416b"
server_key_version: "encryptionkey@0"
I20260924 07:10:05.712849 24727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "28227c9bea4642e6b0710f5a2e737aa6"
format_stamp: "Formatted at 2026-09-24 07:10:05 on dist-test-slave-2p7j"
server_key: "f19546d5d507b862b86179d8f1393eb5"
server_key_iv: "bf44ec318de40ff72e462946eb21416b"
server_key_version: "encryptionkey@0"
I20260924 07:10:05.716768 24727 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:10:05.720166 24742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:05.721709 24727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:10:05.721819 24727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "28227c9bea4642e6b0710f5a2e737aa6"
format_stamp: "Formatted at 2026-09-24 07:10:05 on dist-test-slave-2p7j"
server_key: "f19546d5d507b862b86179d8f1393eb5"
server_key_iv: "bf44ec318de40ff72e462946eb21416b"
server_key_version: "encryptionkey@0"
I20260924 07:10:05.721971 24727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:05.726670 24727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:05.730006 24727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:05.730165 24727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:05.737730 24727 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38089
I20260924 07:10:05.737871 24794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38089 every 8 connection(s)
I20260924 07:10:05.738883 24727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:05.742558 24795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:05.745307 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 24727
I20260924 07:10:05.745426 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:10:05.745700 3940 external_mini_cluster.cc:1428] Setting key dbbf6cffff2d9248924b53f2db13149f
I20260924 07:10:05.750484 24795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28227c9bea4642e6b0710f5a2e737aa6: Bootstrap starting.
I20260924 07:10:05.751591 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:05.747325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46363 (local address 127.0.0.1:38089)
0924 07:10:05.747790 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:05.747805 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:05.747839 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:10:05.748531 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:05.748548 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:05.748826 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:05.749077 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:05.750215 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:05.750222 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:05.750248 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:05.750262 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:05.750335 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:05.750360 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:05.750614 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:05.750710 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 07:10:05.750892 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":152,"threads_started":1}
I20260924 07:10:05.753489 24795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28227c9bea4642e6b0710f5a2e737aa6: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:05.754459 24795 log.cc:824] T 00000000000000000000000000000000 P 28227c9bea4642e6b0710f5a2e737aa6: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:05.754768 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:10:05.756755 24795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28227c9bea4642e6b0710f5a2e737aa6: No bootstrap required, opened a new log
I20260924 07:10:05.758285 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:05.756833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34291 (local address 127.0.0.1:38089)
0924 07:10:05.756939 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:05.756943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:05.757089 (+ 146us) server_negotiation.cc:407] Connection header received
0924 07:10:05.757192 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:05.757195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:05.757229 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:05.757298 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:05.757947 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:05.757951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:05.757954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:05.757965 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:05.757992 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:05.758002 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:05.758072 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:05.758141 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 07:10:05.758177 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:10:05.760413 24795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28227c9bea4642e6b0710f5a2e737aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28227c9bea4642e6b0710f5a2e737aa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38089 } }
I20260924 07:10:05.760641 24795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28227c9bea4642e6b0710f5a2e737aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:05.760699 24795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28227c9bea4642e6b0710f5a2e737aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28227c9bea4642e6b0710f5a2e737aa6, State: Initialized, Role: FOLLOWER
I20260924 07:10:05.761066 24795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28227c9bea4642e6b0710f5a2e737aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28227c9bea4642e6b0710f5a2e737aa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38089 } }
I20260924 07:10:05.761193 24795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28227c9bea4642e6b0710f5a2e737aa6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:05.761230 24795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28227c9bea4642e6b0710f5a2e737aa6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:05.761292 24795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28227c9bea4642e6b0710f5a2e737aa6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:05.762087 24795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28227c9bea4642e6b0710f5a2e737aa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28227c9bea4642e6b0710f5a2e737aa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38089 } }
I20260924 07:10:05.762367 24795 leader_election.cc:302] T 00000000000000000000000000000000 P 28227c9bea4642e6b0710f5a2e737aa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28227c9bea4642e6b0710f5a2e737aa6; no voters:
I20260924 07:10:05.762607 24795 leader_election.cc:288] T 00000000000000000000000000000000 P 28227c9bea4642e6b0710f5a2e737aa6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:10:05.762761 24805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28227c9bea4642e6b0710f5a2e737aa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:05.762939 24805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28227c9bea4642e6b0710f5a2e737aa6 [term 1 LEADER]: Becoming Leader. State: Replica: 28227c9bea4642e6b0710f5a2e737aa6, State: Running, Role: LEADER
I20260924 07:10:05.763607 24805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28227c9bea4642e6b0710f5a2e737aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28227c9bea4642e6b0710f5a2e737aa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38089 } }
I20260924 07:10:05.763844 24795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28227c9bea4642e6b0710f5a2e737aa6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:05.765120 24806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28227c9bea4642e6b0710f5a2e737aa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28227c9bea4642e6b0710f5a2e737aa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28227c9bea4642e6b0710f5a2e737aa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38089 } } }
I20260924 07:10:05.765249 24806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28227c9bea4642e6b0710f5a2e737aa6 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:05.765551 24807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28227c9bea4642e6b0710f5a2e737aa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28227c9bea4642e6b0710f5a2e737aa6. Latest consensus state: current_term: 1 leader_uuid: "28227c9bea4642e6b0710f5a2e737aa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28227c9bea4642e6b0710f5a2e737aa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38089 } } }
I20260924 07:10:05.765646 24807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28227c9bea4642e6b0710f5a2e737aa6 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:05.765918 24815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:05.769166 24815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:05.774381 24815 catalog_manager.cc:1347] Generated new cluster ID: 49aef55dca144f0b9fd5f6db20f74650
I20260924 07:10:05.774472 24815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:05.796744 24815 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:05.797505 24815 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:05.806913 24815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28227c9bea4642e6b0710f5a2e737aa6: Generated new TSK 0
I20260924 07:10:05.807644 24815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:10:05.823446 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 24727
2026-09-24T07:10:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:10:05 dist-test-slave-2p7j krb5kdc[24826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:10:05 dist-test-slave-2p7j krb5kdc[24826](info): set up 2 sockets
Sep 24 07:10:05 dist-test-slave-2p7j krb5kdc[24826](info): commencing operation
krb5kdc: starting...
W20260924 07:10:06.807909 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:10:06 dist-test-slave-2p7j krb5kdc[24826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:10:06.931185 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46871
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:07.041653 24839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:07.041898 24839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:07.041936 24839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:07.046202 24839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:07.046298 24839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:07.046330 24839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:07.046353 24839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:07.046381 24839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:07.051833 24839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:10:07.052868 24839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:07.053812 24839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:07.061767 24844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:07.061923 24847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:07.061923 24845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:07.062032 24839 server_base.cc:1034] running on GCE node
I20260924 07:10:07.062973 24839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:07.064443 24839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:07.065634 24839 hybrid_clock.cc:648] HybridClock initialized: now 1790233807065615 us; error 33 us; skew 500 ppm
Sep 24 07:10:07 dist-test-slave-2p7j krb5kdc[24826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:07.073459 24839 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:10:07.074438 24839 webserver.cc:460] Webserver started at http://127.0.0.1:37113/ using document root <none> and password file <none>
I20260924 07:10:07.074963 24839 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:07.075021 24839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:07.075193 24839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:07.077171 24839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "d4afb5a37f1d43dcabc17a3d73274e0d"
format_stamp: "Formatted at 2026-09-24 07:10:07 on dist-test-slave-2p7j"
server_key: "e25ee5cd22277604d5dbe398b00fc031"
server_key_iv: "d13569df8172d26be8c1bb3f522023e8"
server_key_version: "encryptionkey@0"
I20260924 07:10:07.077682 24839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "d4afb5a37f1d43dcabc17a3d73274e0d"
format_stamp: "Formatted at 2026-09-24 07:10:07 on dist-test-slave-2p7j"
server_key: "e25ee5cd22277604d5dbe398b00fc031"
server_key_iv: "d13569df8172d26be8c1bb3f522023e8"
server_key_version: "encryptionkey@0"
I20260924 07:10:07.081765 24839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:10:07.084198 24854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:07.085417 24839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:10:07.085526 24839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "d4afb5a37f1d43dcabc17a3d73274e0d"
format_stamp: "Formatted at 2026-09-24 07:10:07 on dist-test-slave-2p7j"
server_key: "e25ee5cd22277604d5dbe398b00fc031"
server_key_iv: "d13569df8172d26be8c1bb3f522023e8"
server_key_version: "encryptionkey@0"
I20260924 07:10:07.085628 24839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:07.119235 24839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:07.123044 24839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:07.123231 24839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:07.130261 24839 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46871
I20260924 07:10:07.130294 24906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46871 every 8 connection(s)
I20260924 07:10:07.131160 24839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:07.133877 24907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:07.138373 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 24839
I20260924 07:10:07.138511 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:10:07.138784 3940 external_mini_cluster.cc:1428] Setting key c874cfe7080d5c2efff1c9b29a25ea1b
I20260924 07:10:07.139364 24907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4afb5a37f1d43dcabc17a3d73274e0d: Bootstrap starting.
I20260924 07:10:07.141476 24907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4afb5a37f1d43dcabc17a3d73274e0d: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:07.142285 24907 log.cc:824] T 00000000000000000000000000000000 P d4afb5a37f1d43dcabc17a3d73274e0d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:07.144255 24907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4afb5a37f1d43dcabc17a3d73274e0d: No bootstrap required, opened a new log
Sep 24 07:10:07 dist-test-slave-2p7j krb5kdc[24826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 07:10:07.148313 24907 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4afb5a37f1d43dcabc17a3d73274e0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4afb5a37f1d43dcabc17a3d73274e0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46871 } }
I20260924 07:10:07.148578 24907 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4afb5a37f1d43dcabc17a3d73274e0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:07.148641 24907 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4afb5a37f1d43dcabc17a3d73274e0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4afb5a37f1d43dcabc17a3d73274e0d, State: Initialized, Role: FOLLOWER
I20260924 07:10:07.149091 24907 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4afb5a37f1d43dcabc17a3d73274e0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4afb5a37f1d43dcabc17a3d73274e0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46871 } }
I20260924 07:10:07.149246 24907 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4afb5a37f1d43dcabc17a3d73274e0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:07.149310 24907 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4afb5a37f1d43dcabc17a3d73274e0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:07.149439 24907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4afb5a37f1d43dcabc17a3d73274e0d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:07.150257 24907 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4afb5a37f1d43dcabc17a3d73274e0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4afb5a37f1d43dcabc17a3d73274e0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46871 } }
I20260924 07:10:07.150619 24907 leader_election.cc:302] T 00000000000000000000000000000000 P d4afb5a37f1d43dcabc17a3d73274e0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4afb5a37f1d43dcabc17a3d73274e0d; no voters:
I20260924 07:10:07.150940 24907 leader_election.cc:288] T 00000000000000000000000000000000 P d4afb5a37f1d43dcabc17a3d73274e0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:10:07.151032 24912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4afb5a37f1d43dcabc17a3d73274e0d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:07.151280 24912 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4afb5a37f1d43dcabc17a3d73274e0d [term 1 LEADER]: Becoming Leader. State: Replica: d4afb5a37f1d43dcabc17a3d73274e0d, State: Running, Role: LEADER
I20260924 07:10:07.151669 24912 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4afb5a37f1d43dcabc17a3d73274e0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4afb5a37f1d43dcabc17a3d73274e0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46871 } }
I20260924 07:10:07.152171 24907 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4afb5a37f1d43dcabc17a3d73274e0d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:07.153187 24914 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4afb5a37f1d43dcabc17a3d73274e0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4afb5a37f1d43dcabc17a3d73274e0d. Latest consensus state: current_term: 1 leader_uuid: "d4afb5a37f1d43dcabc17a3d73274e0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4afb5a37f1d43dcabc17a3d73274e0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46871 } } }
I20260924 07:10:07.153307 24914 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4afb5a37f1d43dcabc17a3d73274e0d [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:07.153463 24910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:07.140166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.157:54933 (local address 127.0.0.1:46871)
0924 07:10:07.140550 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:07.140565 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:07.140604 (+ 39us) server_negotiation.cc:407] Connection header received
0924 07:10:07.141236 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:07.141252 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:07.141601 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:07.141869 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:07.142724 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:07.143881 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:07.145017 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:07.145495 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:07.145615 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:07.148396 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:07.148403 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:07.148418 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:07.148448 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:07.150117 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:07.150655 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:07.150660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:07.150665 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:07.150810 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:07.151198 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:07.151201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:07.151203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:07.151736 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:07.152010 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:07.152290 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 07:10:07.152521 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":104,"threads_started":1}
I20260924 07:10:07.153648 24913 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4afb5a37f1d43dcabc17a3d73274e0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4afb5a37f1d43dcabc17a3d73274e0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4afb5a37f1d43dcabc17a3d73274e0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46871 } } }
I20260924 07:10:07.153759 24913 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4afb5a37f1d43dcabc17a3d73274e0d [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:07.154650 24921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:07.156605 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:10:07.157497 24921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:07.163352 24921 catalog_manager.cc:1347] Generated new cluster ID: 7f6de1fc57bf4d858495ce7de730ce1e
I20260924 07:10:07.163453 24921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:07.167893 24910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:07.158700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.157:56333 (local address 127.0.0.1:46871)
0924 07:10:07.158805 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:07.158808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:07.159492 (+ 684us) server_negotiation.cc:407] Connection header received
0924 07:10:07.159616 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:07.159620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:07.159668 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:07.159766 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:07.160449 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:07.161634 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:07.162996 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:07.163381 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:07.163589 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:07.165011 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:07.165015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:07.165017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:07.165038 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:07.166286 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:07.166712 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:07.166715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:07.166716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:07.166783 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:07.167066 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:07.167069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:07.167070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:07.167227 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:07.167417 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:07.167666 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 07:10:07.167751 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 07:10:07.174800 24921 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:07.176090 24921 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:07.182564 24921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4afb5a37f1d43dcabc17a3d73274e0d: Generated new TSK 0
I20260924 07:10:07.183259 24921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:10:07.193562 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 24839
W20260924 07:10:07.198779 24824 connection.cc:535] client connection to 127.0.0.1:46871 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46871): Connection reset by peer (error 104)
2026-09-24T07:10:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:10:07 dist-test-slave-2p7j krb5kdc[24938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:10:07 dist-test-slave-2p7j krb5kdc[24938](info): set up 2 sockets
Sep 24 07:10:07 dist-test-slave-2p7j krb5kdc[24938](info): commencing operation
krb5kdc: starting...
W20260924 07:10:08.164870 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:10:08 dist-test-slave-2p7j krb5kdc[24938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:10:08.286556 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40419
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:08.401453 24951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:08.401815 24951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:08.401882 24951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:08.408010 24951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:08.408119 24951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:08.408146 24951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:08.408172 24951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:08.408201 24951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:08.415442 24951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:10:08.416625 24951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:08.417621 24951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:08.426967 24957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:08.426986 24956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:08.427026 24959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:08.427147 24951 server_base.cc:1034] running on GCE node
I20260924 07:10:08.427815 24951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:08.428979 24951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:08.430301 24951 hybrid_clock.cc:648] HybridClock initialized: now 1790233808430259 us; error 97 us; skew 500 ppm
Sep 24 07:10:08 dist-test-slave-2p7j krb5kdc[24938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:08.433689 24951 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:10:08.434554 24951 webserver.cc:460] Webserver started at http://127.0.0.1:37489/ using document root <none> and password file <none>
I20260924 07:10:08.435077 24951 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:08.435141 24951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:08.435319 24951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:08.437275 24951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "994ecbf37ad141e490f8695e30d62b18"
format_stamp: "Formatted at 2026-09-24 07:10:08 on dist-test-slave-2p7j"
server_key: "475177b224889fe7321397fb47ccb3fe"
server_key_iv: "40472621b08ee76c9dfe64344bec774d"
server_key_version: "encryptionkey@0"
I20260924 07:10:08.437806 24951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "994ecbf37ad141e490f8695e30d62b18"
format_stamp: "Formatted at 2026-09-24 07:10:08 on dist-test-slave-2p7j"
server_key: "475177b224889fe7321397fb47ccb3fe"
server_key_iv: "40472621b08ee76c9dfe64344bec774d"
server_key_version: "encryptionkey@0"
I20260924 07:10:08.441282 24951 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:10:08.444094 24966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:08.445587 24951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:10:08.445837 24951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "994ecbf37ad141e490f8695e30d62b18"
format_stamp: "Formatted at 2026-09-24 07:10:08 on dist-test-slave-2p7j"
server_key: "475177b224889fe7321397fb47ccb3fe"
server_key_iv: "40472621b08ee76c9dfe64344bec774d"
server_key_version: "encryptionkey@0"
I20260924 07:10:08.446072 24951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:08.460420 24951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:08.463671 24951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:08.463829 24951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:08.471410 24951 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40419
I20260924 07:10:08.471678 25018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40419 every 8 connection(s)
I20260924 07:10:08.472347 24951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:08.474041 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 24951
I20260924 07:10:08.474197 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:10:08.474534 3940 external_mini_cluster.cc:1428] Setting key 6d7b5d980ea2b5cd1839bdd16de699d4
I20260924 07:10:08.475291 25019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:08.482100 25019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 994ecbf37ad141e490f8695e30d62b18: Bootstrap starting.
I20260924 07:10:08.483402 25022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:08.476108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.157:36735 (local address 127.0.0.1:40419)
0924 07:10:08.476520 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:08.476535 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:08.476570 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:10:08.477216 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:08.477230 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:08.477548 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:08.477845 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:08.478177 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:08.479169 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:08.480351 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:08.480822 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:08.480966 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:08.481803 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:08.481807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:08.481818 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:08.482196 (+ 378us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:08.482271 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:08.482301 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:08.482405 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:08.482585 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 07:10:08.482712 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":117,"threads_started":1}
I20260924 07:10:08.484297 25019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 994ecbf37ad141e490f8695e30d62b18: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:08.485083 25019 log.cc:824] T 00000000000000000000000000000000 P 994ecbf37ad141e490f8695e30d62b18: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:08.486474 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:10:08.486853 25019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 994ecbf37ad141e490f8695e30d62b18: No bootstrap required, opened a new log
I20260924 07:10:08.490976 25019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 994ecbf37ad141e490f8695e30d62b18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "994ecbf37ad141e490f8695e30d62b18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40419 } }
I20260924 07:10:08.491212 25019 raft_consensus.cc:374] T 00000000000000000000000000000000 P 994ecbf37ad141e490f8695e30d62b18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:08.491262 25019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 994ecbf37ad141e490f8695e30d62b18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 994ecbf37ad141e490f8695e30d62b18, State: Initialized, Role: FOLLOWER
I20260924 07:10:08.491726 25019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 994ecbf37ad141e490f8695e30d62b18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "994ecbf37ad141e490f8695e30d62b18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40419 } }
I20260924 07:10:08.491864 25019 raft_consensus.cc:388] T 00000000000000000000000000000000 P 994ecbf37ad141e490f8695e30d62b18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:08.491925 25019 raft_consensus.cc:482] T 00000000000000000000000000000000 P 994ecbf37ad141e490f8695e30d62b18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:08.492023 25019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 994ecbf37ad141e490f8695e30d62b18 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:08.493067 25019 raft_consensus.cc:504] T 00000000000000000000000000000000 P 994ecbf37ad141e490f8695e30d62b18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "994ecbf37ad141e490f8695e30d62b18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40419 } }
I20260924 07:10:08.493171 25022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:08.488646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.157:34793 (local address 127.0.0.1:40419)
0924 07:10:08.488753 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:08.488757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:08.488874 (+ 117us) server_negotiation.cc:407] Connection header received
0924 07:10:08.488963 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:08.488966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:08.489012 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:08.489098 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:08.489296 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:08.490044 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:08.491171 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:08.491528 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:08.491634 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:08.492693 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:08.492697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:08.492699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:08.492711 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:08.492740 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:08.492753 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:08.492893 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:08.492981 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 07:10:08.493018 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:10:08.493432 25019 leader_election.cc:302] T 00000000000000000000000000000000 P 994ecbf37ad141e490f8695e30d62b18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 994ecbf37ad141e490f8695e30d62b18; no voters:
I20260924 07:10:08.493746 25019 leader_election.cc:288] T 00000000000000000000000000000000 P 994ecbf37ad141e490f8695e30d62b18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:10:08.493921 25029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 994ecbf37ad141e490f8695e30d62b18 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:08.494313 25029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 994ecbf37ad141e490f8695e30d62b18 [term 1 LEADER]: Becoming Leader. State: Replica: 994ecbf37ad141e490f8695e30d62b18, State: Running, Role: LEADER
I20260924 07:10:08.494586 25029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 994ecbf37ad141e490f8695e30d62b18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "994ecbf37ad141e490f8695e30d62b18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40419 } }
I20260924 07:10:08.494946 25019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 994ecbf37ad141e490f8695e30d62b18 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:08.495957 25030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 994ecbf37ad141e490f8695e30d62b18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "994ecbf37ad141e490f8695e30d62b18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "994ecbf37ad141e490f8695e30d62b18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40419 } } }
I20260924 07:10:08.496078 25030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 994ecbf37ad141e490f8695e30d62b18 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:08.496495 25031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 994ecbf37ad141e490f8695e30d62b18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 994ecbf37ad141e490f8695e30d62b18. Latest consensus state: current_term: 1 leader_uuid: "994ecbf37ad141e490f8695e30d62b18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "994ecbf37ad141e490f8695e30d62b18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40419 } } }
I20260924 07:10:08.496585 25031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 994ecbf37ad141e490f8695e30d62b18 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:08.496835 25038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:08.499418 25038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:08.505388 25038 catalog_manager.cc:1347] Generated new cluster ID: d3b1f7a15f3e46ae9a1a05812665f474
I20260924 07:10:08.505452 25038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:08.524830 25038 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:08.526011 25038 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:08.535673 25038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 994ecbf37ad141e490f8695e30d62b18: Generated new TSK 0
I20260924 07:10:08.536437 25038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:10:08.551764 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 24951
W20260924 07:10:08.558475 24936 connection.cc:535] client connection to 127.0.0.1:40419 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40419): Connection reset by peer (error 104)
2026-09-24T07:10:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:10:08 dist-test-slave-2p7j krb5kdc[25050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:10:08 dist-test-slave-2p7j krb5kdc[25050](info): set up 2 sockets
Sep 24 07:10:08 dist-test-slave-2p7j krb5kdc[25050](info): commencing operation
krb5kdc: starting...
W20260924 07:10:09.516644 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:10:09 dist-test-slave-2p7j krb5kdc[25050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:10:09.651273 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33981
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:09.780503 25063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:09.780856 25063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:09.780920 25063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:09.785311 25063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:09.785406 25063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:09.785434 25063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:09.785458 25063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:09.785481 25063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:09.791936 25063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:10:09.793367 25063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:09.794664 25063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:09.804780 25068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:09.804814 25069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:09.804994 25063 server_base.cc:1034] running on GCE node
W20260924 07:10:09.804841 25071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:09.806067 25063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:09.807482 25063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:09.808714 25063 hybrid_clock.cc:648] HybridClock initialized: now 1790233809808685 us; error 43 us; skew 500 ppm
Sep 24 07:10:09 dist-test-slave-2p7j krb5kdc[25050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:09.811908 25063 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:10:09.813107 25063 webserver.cc:460] Webserver started at http://127.0.0.1:38633/ using document root <none> and password file <none>
I20260924 07:10:09.813757 25063 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:09.813830 25063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:09.814071 25063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:09.816603 25063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "7b65f96b12f144849e98b0e37938256c"
format_stamp: "Formatted at 2026-09-24 07:10:09 on dist-test-slave-2p7j"
server_key: "c0b40c810b9cc02d58892ddd1997a6db"
server_key_iv: "e358b1b7602fd97a29802486cfe03484"
server_key_version: "encryptionkey@0"
I20260924 07:10:09.817271 25063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "7b65f96b12f144849e98b0e37938256c"
format_stamp: "Formatted at 2026-09-24 07:10:09 on dist-test-slave-2p7j"
server_key: "c0b40c810b9cc02d58892ddd1997a6db"
server_key_iv: "e358b1b7602fd97a29802486cfe03484"
server_key_version: "encryptionkey@0"
I20260924 07:10:09.821906 25063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:10:09.824930 25078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:09.826519 25063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:10:09.826680 25063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "7b65f96b12f144849e98b0e37938256c"
format_stamp: "Formatted at 2026-09-24 07:10:09 on dist-test-slave-2p7j"
server_key: "c0b40c810b9cc02d58892ddd1997a6db"
server_key_iv: "e358b1b7602fd97a29802486cfe03484"
server_key_version: "encryptionkey@0"
I20260924 07:10:09.826813 25063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:09.832239 25063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:09.835806 25063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:09.836007 25063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:09.844748 25063 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33981
I20260924 07:10:09.844810 25130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33981 every 8 connection(s)
I20260924 07:10:09.845979 25063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:09.848784 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 25063
I20260924 07:10:09.848920 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:10:09.849246 3940 external_mini_cluster.cc:1428] Setting key ea9e26ab21b6ea0772a307f733bd8cf1
I20260924 07:10:09.849509 25131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:09.856600 25134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.850948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.157:49207 (local address 127.0.0.1:33981)
0924 07:10:09.851627 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.851649 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.851698 (+ 49us) server_negotiation.cc:407] Connection header received
0924 07:10:09.852915 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.852944 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.853370 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.853958 (+ 588us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:09.854923 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:09.854929 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:09.854950 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:09.854963 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:09.855032 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:09.855056 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:09.855316 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:09.855406 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 07:10:09.855653 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":177,"threads_started":1}
I20260924 07:10:09.856768 25131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b65f96b12f144849e98b0e37938256c: Bootstrap starting.
I20260924 07:10:09.859071 25131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b65f96b12f144849e98b0e37938256c: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:09.859863 25131 log.cc:824] T 00000000000000000000000000000000 P 7b65f96b12f144849e98b0e37938256c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:09.860330 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:10:09.862360 25131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b65f96b12f144849e98b0e37938256c: No bootstrap required, opened a new log
I20260924 07:10:09.864138 25134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.862306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.157:42847 (local address 127.0.0.1:33981)
0924 07:10:09.862441 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.862446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.862604 (+ 158us) server_negotiation.cc:407] Connection header received
0924 07:10:09.862822 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.862825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.862869 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.862965 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:09.863778 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:09.863781 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:09.863783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:09.863794 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:09.863821 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:09.863831 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:09.863908 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:09.863992 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 07:10:09.864028 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 07:10:09.867317 25131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b65f96b12f144849e98b0e37938256c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b65f96b12f144849e98b0e37938256c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33981 } }
I20260924 07:10:09.867669 25131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b65f96b12f144849e98b0e37938256c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:09.867739 25131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b65f96b12f144849e98b0e37938256c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b65f96b12f144849e98b0e37938256c, State: Initialized, Role: FOLLOWER
I20260924 07:10:09.868249 25131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b65f96b12f144849e98b0e37938256c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b65f96b12f144849e98b0e37938256c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33981 } }
I20260924 07:10:09.868398 25131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b65f96b12f144849e98b0e37938256c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:09.868467 25131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b65f96b12f144849e98b0e37938256c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:09.868575 25131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b65f96b12f144849e98b0e37938256c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:09.869683 25131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b65f96b12f144849e98b0e37938256c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b65f96b12f144849e98b0e37938256c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33981 } }
I20260924 07:10:09.870091 25131 leader_election.cc:302] T 00000000000000000000000000000000 P 7b65f96b12f144849e98b0e37938256c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b65f96b12f144849e98b0e37938256c; no voters:
I20260924 07:10:09.870464 25131 leader_election.cc:288] T 00000000000000000000000000000000 P 7b65f96b12f144849e98b0e37938256c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:10:09.870647 25141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b65f96b12f144849e98b0e37938256c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:09.870941 25141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b65f96b12f144849e98b0e37938256c [term 1 LEADER]: Becoming Leader. State: Replica: 7b65f96b12f144849e98b0e37938256c, State: Running, Role: LEADER
I20260924 07:10:09.871346 25141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b65f96b12f144849e98b0e37938256c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b65f96b12f144849e98b0e37938256c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33981 } }
I20260924 07:10:09.872052 25131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b65f96b12f144849e98b0e37938256c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:09.873229 25143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b65f96b12f144849e98b0e37938256c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b65f96b12f144849e98b0e37938256c. Latest consensus state: current_term: 1 leader_uuid: "7b65f96b12f144849e98b0e37938256c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b65f96b12f144849e98b0e37938256c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33981 } } }
I20260924 07:10:09.873390 25143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b65f96b12f144849e98b0e37938256c [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:09.873698 25142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b65f96b12f144849e98b0e37938256c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b65f96b12f144849e98b0e37938256c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b65f96b12f144849e98b0e37938256c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33981 } } }
I20260924 07:10:09.873811 25142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b65f96b12f144849e98b0e37938256c [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:09.874531 25150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:09.877373 25150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:09.882691 25150 catalog_manager.cc:1347] Generated new cluster ID: e766709524d441e59127522d17956dbe
I20260924 07:10:09.882762 25150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:09.894191 25150 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:09.894678 25150 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:09.906025 25150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b65f96b12f144849e98b0e37938256c: Generated new TSK 0
I20260924 07:10:09.906780 25150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:10:09.914989 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 25063
2026-09-24T07:10:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:10:09 dist-test-slave-2p7j krb5kdc[25162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:10:09 dist-test-slave-2p7j krb5kdc[25162](info): set up 2 sockets
Sep 24 07:10:09 dist-test-slave-2p7j krb5kdc[25162](info): commencing operation
krb5kdc: starting...
W20260924 07:10:10.888859 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:10:10 dist-test-slave-2p7j krb5kdc[25162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:10:11.028825 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45541
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:11.144879 25175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:11.145112 25175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:11.145150 25175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:11.149120 25175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:11.149194 25175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:11.149217 25175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:11.149236 25175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:11.149264 25175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:11.154528 25175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45541
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:10:11.155534 25175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:11.156518 25175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:11.164862 25180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:11.164849 25183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:11.164878 25181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:11.164966 25175 server_base.cc:1034] running on GCE node
I20260924 07:10:11.165886 25175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:11.166962 25175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:11.168131 25175 hybrid_clock.cc:648] HybridClock initialized: now 1790233811168113 us; error 32 us; skew 500 ppm
Sep 24 07:10:11 dist-test-slave-2p7j krb5kdc[25162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:11.171221 25175 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:10:11.172211 25175 webserver.cc:460] Webserver started at http://127.0.0.1:35559/ using document root <none> and password file <none>
I20260924 07:10:11.172716 25175 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:11.172775 25175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:11.172926 25175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:11.174867 25175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "064d4ec810054b99b0f048876fa4cd3a"
format_stamp: "Formatted at 2026-09-24 07:10:11 on dist-test-slave-2p7j"
server_key: "8754f49c3c80373cc601df5532aec7a2"
server_key_iv: "e120b5f5e57a7d10b0c2687f70672c23"
server_key_version: "encryptionkey@0"
I20260924 07:10:11.175354 25175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "064d4ec810054b99b0f048876fa4cd3a"
format_stamp: "Formatted at 2026-09-24 07:10:11 on dist-test-slave-2p7j"
server_key: "8754f49c3c80373cc601df5532aec7a2"
server_key_iv: "e120b5f5e57a7d10b0c2687f70672c23"
server_key_version: "encryptionkey@0"
I20260924 07:10:11.178833 25175 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:10:11.181178 25190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:11.182448 25175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:10:11.182548 25175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "064d4ec810054b99b0f048876fa4cd3a"
format_stamp: "Formatted at 2026-09-24 07:10:11 on dist-test-slave-2p7j"
server_key: "8754f49c3c80373cc601df5532aec7a2"
server_key_iv: "e120b5f5e57a7d10b0c2687f70672c23"
server_key_version: "encryptionkey@0"
I20260924 07:10:11.182650 25175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:11.212073 25175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:11.215052 25175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:11.215224 25175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:11.222465 25175 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45541
I20260924 07:10:11.222506 25242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45541 every 8 connection(s)
I20260924 07:10:11.223435 25175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:11.225510 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 25175
I20260924 07:10:11.225615 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:10:11.225898 3940 external_mini_cluster.cc:1428] Setting key ad7edeb616aa1d16ec2bf57f1884ed88
I20260924 07:10:11.226768 25243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:11.234800 25243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 064d4ec810054b99b0f048876fa4cd3a: Bootstrap starting.
Sep 24 07:10:11 dist-test-slave-2p7j krb5kdc[25162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 07:10:11.237607 25243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 064d4ec810054b99b0f048876fa4cd3a: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:11.238420 25243 log.cc:824] T 00000000000000000000000000000000 P 064d4ec810054b99b0f048876fa4cd3a: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:11.240031 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:11.227655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.157:57717 (local address 127.0.0.1:45541)
0924 07:10:11.228230 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:11.228253 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:11.228289 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:10:11.229019 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:11.229038 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:11.229397 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:11.229706 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:11.230457 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:11.231449 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:11.232262 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:11.232713 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:11.232860 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:11.235444 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:11.235448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:11.235460 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:11.235486 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:11.237144 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:11.237589 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:11.237593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:11.237598 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:11.237722 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:11.238078 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:11.238080 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:11.238082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:11.238437 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:11.238622 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:11.238924 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 07:10:11.239120 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":301,"threads_started":1}
I20260924 07:10:11.240520 25243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 064d4ec810054b99b0f048876fa4cd3a: No bootstrap required, opened a new log
I20260924 07:10:11.243335 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:10:11.244232 25243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 064d4ec810054b99b0f048876fa4cd3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "064d4ec810054b99b0f048876fa4cd3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45541 } }
I20260924 07:10:11.244488 25243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 064d4ec810054b99b0f048876fa4cd3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:11.244570 25243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 064d4ec810054b99b0f048876fa4cd3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 064d4ec810054b99b0f048876fa4cd3a, State: Initialized, Role: FOLLOWER
I20260924 07:10:11.244961 25243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 064d4ec810054b99b0f048876fa4cd3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "064d4ec810054b99b0f048876fa4cd3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45541 } }
I20260924 07:10:11.245085 25243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 064d4ec810054b99b0f048876fa4cd3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:11.245141 25243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 064d4ec810054b99b0f048876fa4cd3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:11.245246 25243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 064d4ec810054b99b0f048876fa4cd3a [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:11.246235 25243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 064d4ec810054b99b0f048876fa4cd3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "064d4ec810054b99b0f048876fa4cd3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45541 } }
I20260924 07:10:11.246589 25243 leader_election.cc:302] T 00000000000000000000000000000000 P 064d4ec810054b99b0f048876fa4cd3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 064d4ec810054b99b0f048876fa4cd3a; no voters:
I20260924 07:10:11.246886 25243 leader_election.cc:288] T 00000000000000000000000000000000 P 064d4ec810054b99b0f048876fa4cd3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:10:11.247210 25253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 064d4ec810054b99b0f048876fa4cd3a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:11.247562 25253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 064d4ec810054b99b0f048876fa4cd3a [term 1 LEADER]: Becoming Leader. State: Replica: 064d4ec810054b99b0f048876fa4cd3a, State: Running, Role: LEADER
I20260924 07:10:11.248155 25243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 064d4ec810054b99b0f048876fa4cd3a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:11.248114 25253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 064d4ec810054b99b0f048876fa4cd3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "064d4ec810054b99b0f048876fa4cd3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45541 } }
I20260924 07:10:11.249827 25255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 064d4ec810054b99b0f048876fa4cd3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "064d4ec810054b99b0f048876fa4cd3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "064d4ec810054b99b0f048876fa4cd3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45541 } } }
I20260924 07:10:11.249984 25255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 064d4ec810054b99b0f048876fa4cd3a [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:11.250272 25254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 064d4ec810054b99b0f048876fa4cd3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 064d4ec810054b99b0f048876fa4cd3a. Latest consensus state: current_term: 1 leader_uuid: "064d4ec810054b99b0f048876fa4cd3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "064d4ec810054b99b0f048876fa4cd3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45541 } } }
I20260924 07:10:11.250811 25254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 064d4ec810054b99b0f048876fa4cd3a [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:11.250427 25268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:11.253777 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:11.245210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.157:38327 (local address 127.0.0.1:45541)
0924 07:10:11.245338 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:11.245342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:11.245478 (+ 136us) server_negotiation.cc:407] Connection header received
0924 07:10:11.245544 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:11.245547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:11.245590 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:11.245663 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:11.246225 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:11.246941 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:11.248111 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:11.248526 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:11.248633 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:11.250176 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:11.250180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:11.250183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:11.250205 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:11.251890 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:11.252444 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:11.252448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:11.252449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:11.252530 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:11.252860 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:11.252863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:11.252864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:11.253058 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:11.253197 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:11.253508 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 07:10:11.253608 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:10:11.254034 25268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:11.260093 25268 catalog_manager.cc:1347] Generated new cluster ID: c6e53061ccca4fdfaac9a7ce0de4676a
I20260924 07:10:11.260222 25268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:11.272233 25268 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:11.273392 25268 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:11.293807 25268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 064d4ec810054b99b0f048876fa4cd3a: Generated new TSK 0
I20260924 07:10:11.294857 25268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:10:11.300635 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 25175
W20260924 07:10:11.305840 25160 connection.cc:535] client connection to 127.0.0.1:45541 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45541): Connection reset by peer (error 104)
2026-09-24T07:10:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:10:11 dist-test-slave-2p7j krb5kdc[25274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:10:11 dist-test-slave-2p7j krb5kdc[25274](info): set up 2 sockets
Sep 24 07:10:11 dist-test-slave-2p7j krb5kdc[25274](info): commencing operation
krb5kdc: starting...
W20260924 07:10:12.268896 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:10:12 dist-test-slave-2p7j krb5kdc[25274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:10:12.394976 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34751
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:12.508477 25287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:12.508741 25287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:12.508903 25287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:12.514674 25287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:12.514806 25287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:12.514865 25287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:12.514896 25287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:12.514914 25287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:12.521716 25287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34751
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:10:12.523152 25287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:12.524878 25287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:12.533241 25293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:12.533229 25292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:12.533535 25287 server_base.cc:1034] running on GCE node
W20260924 07:10:12.533567 25295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:12.534096 25287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:12.535082 25287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:12.536271 25287 hybrid_clock.cc:648] HybridClock initialized: now 1790233812536241 us; error 51 us; skew 500 ppm
Sep 24 07:10:12 dist-test-slave-2p7j krb5kdc[25274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:12.544499 25287 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:10:12.545320 25287 webserver.cc:460] Webserver started at http://127.0.0.1:32875/ using document root <none> and password file <none>
I20260924 07:10:12.545815 25287 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:12.545888 25287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:12.546061 25287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:12.548020 25287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "229de574d9b34ed3b795021e61e5533e"
format_stamp: "Formatted at 2026-09-24 07:10:12 on dist-test-slave-2p7j"
server_key: "ae0429b675cc103942e8548f9675e52d"
server_key_iv: "7b1407ad96199e3738e10c6e7e4a249b"
server_key_version: "encryptionkey@0"
I20260924 07:10:12.548512 25287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "229de574d9b34ed3b795021e61e5533e"
format_stamp: "Formatted at 2026-09-24 07:10:12 on dist-test-slave-2p7j"
server_key: "ae0429b675cc103942e8548f9675e52d"
server_key_iv: "7b1407ad96199e3738e10c6e7e4a249b"
server_key_version: "encryptionkey@0"
I20260924 07:10:12.551894 25287 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 07:10:12.554759 25302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:12.556514 25287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:10:12.556707 25287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "229de574d9b34ed3b795021e61e5533e"
format_stamp: "Formatted at 2026-09-24 07:10:12 on dist-test-slave-2p7j"
server_key: "ae0429b675cc103942e8548f9675e52d"
server_key_iv: "7b1407ad96199e3738e10c6e7e4a249b"
server_key_version: "encryptionkey@0"
I20260924 07:10:12.556840 25287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:12.579493 25287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:12.583530 25287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:12.583730 25287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:12.590790 25287 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34751
I20260924 07:10:12.590868 25354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34751 every 8 connection(s)
I20260924 07:10:12.591774 25287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:12.595882 25355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:12.601735 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 25287
I20260924 07:10:12.601861 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:10:12.602123 3940 external_mini_cluster.cc:1428] Setting key 842e039c5fe63a1368c27ea5bc5fcf07
I20260924 07:10:12.603641 25355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 229de574d9b34ed3b795021e61e5533e: Bootstrap starting.
I20260924 07:10:12.605914 25355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 229de574d9b34ed3b795021e61e5533e: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:12.606623 25355 log.cc:824] T 00000000000000000000000000000000 P 229de574d9b34ed3b795021e61e5533e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:12.608490 25355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 229de574d9b34ed3b795021e61e5533e: No bootstrap required, opened a new log
I20260924 07:10:12.611030 25358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:12.603574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.157:49115 (local address 127.0.0.1:34751)
0924 07:10:12.604088 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:12.604103 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:12.604142 (+ 39us) server_negotiation.cc:407] Connection header received
0924 07:10:12.604843 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:12.604861 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:12.605200 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:12.605533 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:12.605926 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:12.606951 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:12.607834 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:12.608277 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:12.608428 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:12.609223 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:12.609228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:12.609240 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:12.609640 (+ 400us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:12.609693 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:12.609716 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:12.609830 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:12.610006 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 07:10:12.610132 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":137,"threads_started":1}
I20260924 07:10:12.612658 25355 raft_consensus.cc:348] T 00000000000000000000000000000000 P 229de574d9b34ed3b795021e61e5533e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229de574d9b34ed3b795021e61e5533e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34751 } }
I20260924 07:10:12.613023 25355 raft_consensus.cc:374] T 00000000000000000000000000000000 P 229de574d9b34ed3b795021e61e5533e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:12.613092 25355 raft_consensus.cc:729] T 00000000000000000000000000000000 P 229de574d9b34ed3b795021e61e5533e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 229de574d9b34ed3b795021e61e5533e, State: Initialized, Role: FOLLOWER
I20260924 07:10:12.613662 25355 consensus_queue.cc:260] T 00000000000000000000000000000000 P 229de574d9b34ed3b795021e61e5533e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229de574d9b34ed3b795021e61e5533e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34751 } }
I20260924 07:10:12.613843 25355 raft_consensus.cc:388] T 00000000000000000000000000000000 P 229de574d9b34ed3b795021e61e5533e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:12.614020 25355 raft_consensus.cc:482] T 00000000000000000000000000000000 P 229de574d9b34ed3b795021e61e5533e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:12.614089 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:10:12.614171 25355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 229de574d9b34ed3b795021e61e5533e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:12.615589 25355 raft_consensus.cc:504] T 00000000000000000000000000000000 P 229de574d9b34ed3b795021e61e5533e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229de574d9b34ed3b795021e61e5533e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34751 } }
I20260924 07:10:12.616096 25355 leader_election.cc:302] T 00000000000000000000000000000000 P 229de574d9b34ed3b795021e61e5533e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 229de574d9b34ed3b795021e61e5533e; no voters:
I20260924 07:10:12.616508 25355 leader_election.cc:288] T 00000000000000000000000000000000 P 229de574d9b34ed3b795021e61e5533e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:10:12.616580 25365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 229de574d9b34ed3b795021e61e5533e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:12.616816 25365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 229de574d9b34ed3b795021e61e5533e [term 1 LEADER]: Becoming Leader. State: Replica: 229de574d9b34ed3b795021e61e5533e, State: Running, Role: LEADER
I20260924 07:10:12.617224 25365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 229de574d9b34ed3b795021e61e5533e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229de574d9b34ed3b795021e61e5533e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34751 } }
I20260924 07:10:12.618180 25355 sys_catalog.cc:564] T 00000000000000000000000000000000 P 229de574d9b34ed3b795021e61e5533e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:12.618435 25367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229de574d9b34ed3b795021e61e5533e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "229de574d9b34ed3b795021e61e5533e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229de574d9b34ed3b795021e61e5533e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34751 } } }
I20260924 07:10:12.618561 25367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229de574d9b34ed3b795021e61e5533e [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:12.619145 25366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229de574d9b34ed3b795021e61e5533e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 229de574d9b34ed3b795021e61e5533e. Latest consensus state: current_term: 1 leader_uuid: "229de574d9b34ed3b795021e61e5533e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229de574d9b34ed3b795021e61e5533e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34751 } } }
I20260924 07:10:12.619226 25366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229de574d9b34ed3b795021e61e5533e [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:12.619622 25369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:12.621286 25358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:12.616088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.157:50055 (local address 127.0.0.1:34751)
0924 07:10:12.616184 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:12.616188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:12.616716 (+ 528us) server_negotiation.cc:407] Connection header received
0924 07:10:12.616934 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:12.616938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:12.616998 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:12.617049 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:12.617505 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:12.618268 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:12.619483 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:12.619817 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:12.619911 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:12.620883 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:12.620886 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:12.620889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:12.620902 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:12.620924 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:12.620934 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:12.621004 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:12.621132 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 07:10:12.621164 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 07:10:12.623314 25369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:12.629647 25369 catalog_manager.cc:1347] Generated new cluster ID: ac090ec54ef04f33acc6872a2c823603
I20260924 07:10:12.629709 25369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:12.635710 25369 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:12.636883 25369 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:12.648892 25369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 229de574d9b34ed3b795021e61e5533e: Generated new TSK 0
I20260924 07:10:12.649546 25369 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:10:12.661427 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 25287
W20260924 07:10:12.666345 25272 connection.cc:535] client connection to 127.0.0.1:34751 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34751): Connection reset by peer (error 104)
2026-09-24T07:10:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:10:12 dist-test-slave-2p7j krb5kdc[25386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:10:12 dist-test-slave-2p7j krb5kdc[25386](info): set up 2 sockets
Sep 24 07:10:12 dist-test-slave-2p7j krb5kdc[25386](info): commencing operation
krb5kdc: starting...
W20260924 07:10:13.644404 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:10:13 dist-test-slave-2p7j krb5kdc[25386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:10:13.767639 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36013
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:13.874387 25399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:13.874624 25399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:13.874663 25399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:13.879172 25399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:13.879292 25399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:13.879316 25399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:13.879335 25399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:13.879352 25399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:13.886361 25399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36013
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:10:13.887396 25399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:13.888417 25399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:13.897464 25407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:13.897542 25404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:13.897542 25405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:13.897820 25399 server_base.cc:1034] running on GCE node
I20260924 07:10:13.898404 25399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:13.899487 25399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:13.900702 25399 hybrid_clock.cc:648] HybridClock initialized: now 1790233813900682 us; error 34 us; skew 500 ppm
Sep 24 07:10:13 dist-test-slave-2p7j krb5kdc[25386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:13.904332 25399 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:10:13.905431 25399 webserver.cc:460] Webserver started at http://127.0.0.1:41451/ using document root <none> and password file <none>
I20260924 07:10:13.905963 25399 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:13.906031 25399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:13.906214 25399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:13.908234 25399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "5a72027d65ba4c16b8ddaf3a73d4b254"
format_stamp: "Formatted at 2026-09-24 07:10:13 on dist-test-slave-2p7j"
server_key: "fce326ce0cb52d26a978d2b4138a9010"
server_key_iv: "2b7a3d214cf11428e1d4e648d827517e"
server_key_version: "encryptionkey@0"
I20260924 07:10:13.908789 25399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "5a72027d65ba4c16b8ddaf3a73d4b254"
format_stamp: "Formatted at 2026-09-24 07:10:13 on dist-test-slave-2p7j"
server_key: "fce326ce0cb52d26a978d2b4138a9010"
server_key_iv: "2b7a3d214cf11428e1d4e648d827517e"
server_key_version: "encryptionkey@0"
I20260924 07:10:13.912842 25399 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 07:10:13.915208 25414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:13.916589 25399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:10:13.916703 25399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "5a72027d65ba4c16b8ddaf3a73d4b254"
format_stamp: "Formatted at 2026-09-24 07:10:13 on dist-test-slave-2p7j"
server_key: "fce326ce0cb52d26a978d2b4138a9010"
server_key_iv: "2b7a3d214cf11428e1d4e648d827517e"
server_key_version: "encryptionkey@0"
I20260924 07:10:13.916792 25399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:13.921028 25399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:13.923859 25399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:13.924007 25399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:13.930480 25399 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36013
I20260924 07:10:13.930601 25466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36013 every 8 connection(s)
I20260924 07:10:13.931319 25399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:13.933580 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 25399
I20260924 07:10:13.933700 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:10:13.933964 25467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:13.933974 3940 external_mini_cluster.cc:1428] Setting key d6c90ce4269f070c8352f89e39a0ba3a
I20260924 07:10:13.938793 25470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:13.935110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.157:56037 (local address 127.0.0.1:36013)
0924 07:10:13.935489 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:13.935524 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:13.935563 (+ 39us) server_negotiation.cc:407] Connection header received
0924 07:10:13.936436 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:13.936454 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:13.936703 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:13.936982 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:13.937704 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:13.937708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:13.937724 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:13.937734 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:13.937784 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:13.937803 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:13.937899 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:13.938002 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 07:10:13.938134 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":122,"threads_started":1}
I20260924 07:10:13.940218 25467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a72027d65ba4c16b8ddaf3a73d4b254: Bootstrap starting.
I20260924 07:10:13.941718 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:10:13.942551 25467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a72027d65ba4c16b8ddaf3a73d4b254: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:13.943532 25467 log.cc:824] T 00000000000000000000000000000000 P 5a72027d65ba4c16b8ddaf3a73d4b254: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:13.945413 25470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:13.944228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.157:42229 (local address 127.0.0.1:36013)
0924 07:10:13.944349 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:13.944353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:13.944365 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:10:13.944395 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:13.944397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:13.944436 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:13.944496 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:13.945084 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:13.945088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:13.945090 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:13.945101 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:13.945126 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:13.945136 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:13.945202 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:13.945274 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 07:10:13.945309 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:10:13.946228 25467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a72027d65ba4c16b8ddaf3a73d4b254: No bootstrap required, opened a new log
I20260924 07:10:13.950004 25467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a72027d65ba4c16b8ddaf3a73d4b254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a72027d65ba4c16b8ddaf3a73d4b254" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36013 } }
I20260924 07:10:13.950227 25467 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a72027d65ba4c16b8ddaf3a73d4b254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:13.950260 25467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a72027d65ba4c16b8ddaf3a73d4b254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a72027d65ba4c16b8ddaf3a73d4b254, State: Initialized, Role: FOLLOWER
I20260924 07:10:13.950613 25467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a72027d65ba4c16b8ddaf3a73d4b254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a72027d65ba4c16b8ddaf3a73d4b254" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36013 } }
I20260924 07:10:13.950718 25467 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a72027d65ba4c16b8ddaf3a73d4b254 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:13.950754 25467 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a72027d65ba4c16b8ddaf3a73d4b254 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:13.950819 25467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a72027d65ba4c16b8ddaf3a73d4b254 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:13.951637 25467 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a72027d65ba4c16b8ddaf3a73d4b254 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a72027d65ba4c16b8ddaf3a73d4b254" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36013 } }
I20260924 07:10:13.951920 25467 leader_election.cc:302] T 00000000000000000000000000000000 P 5a72027d65ba4c16b8ddaf3a73d4b254 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a72027d65ba4c16b8ddaf3a73d4b254; no voters:
I20260924 07:10:13.952162 25467 leader_election.cc:288] T 00000000000000000000000000000000 P 5a72027d65ba4c16b8ddaf3a73d4b254 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:10:13.952255 25477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a72027d65ba4c16b8ddaf3a73d4b254 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:13.952436 25477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a72027d65ba4c16b8ddaf3a73d4b254 [term 1 LEADER]: Becoming Leader. State: Replica: 5a72027d65ba4c16b8ddaf3a73d4b254, State: Running, Role: LEADER
I20260924 07:10:13.952760 25477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a72027d65ba4c16b8ddaf3a73d4b254 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a72027d65ba4c16b8ddaf3a73d4b254" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36013 } }
I20260924 07:10:13.953233 25467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a72027d65ba4c16b8ddaf3a73d4b254 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:13.954198 25479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a72027d65ba4c16b8ddaf3a73d4b254 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a72027d65ba4c16b8ddaf3a73d4b254" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a72027d65ba4c16b8ddaf3a73d4b254" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36013 } } }
I20260924 07:10:13.954324 25479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a72027d65ba4c16b8ddaf3a73d4b254 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:13.954677 25486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:13.954663 25478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a72027d65ba4c16b8ddaf3a73d4b254 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a72027d65ba4c16b8ddaf3a73d4b254. Latest consensus state: current_term: 1 leader_uuid: "5a72027d65ba4c16b8ddaf3a73d4b254" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a72027d65ba4c16b8ddaf3a73d4b254" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36013 } } }
I20260924 07:10:13.955258 25478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a72027d65ba4c16b8ddaf3a73d4b254 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:13.957374 25486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:13.962930 25486 catalog_manager.cc:1347] Generated new cluster ID: 84503632e63b4281aa4e841126abf5ba
I20260924 07:10:13.963039 25486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:13.984190 25486 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:13.984537 25486 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:13.997712 25486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a72027d65ba4c16b8ddaf3a73d4b254: Generated new TSK 0
I20260924 07:10:13.998481 25486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:10:14.010851 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 25399
2026-09-24T07:10:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1349 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:10:14 dist-test-slave-2p7j krb5kdc[25498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:10:14 dist-test-slave-2p7j krb5kdc[25498](info): set up 2 sockets
Sep 24 07:10:14 dist-test-slave-2p7j krb5kdc[25498](info): commencing operation
krb5kdc: starting...
W20260924 07:10:15.036755 3940 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:10:15 dist-test-slave-2p7j krb5kdc[25498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:10:15.171131 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:37315
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:44531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:37315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:15.283820 25511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:15.284130 25511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:15.284195 25511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:15.289232 25511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:15.289311 25511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:15.289341 25511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:15.289363 25511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:15.289391 25511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:15.295236 25511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:44531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:37315
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:37315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:10:15.296336 25511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:15.297329 25511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:15.306347 25517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:15.306365 25516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:15.306419 25511 server_base.cc:1034] running on GCE node
W20260924 07:10:15.306370 25519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:15.307047 25511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:15.308149 25511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:15.309342 25511 hybrid_clock.cc:648] HybridClock initialized: now 1790233815309331 us; error 49 us; skew 500 ppm
Sep 24 07:10:15 dist-test-slave-2p7j krb5kdc[25498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233815, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:15.312115 25511 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:10:15.313007 25511 webserver.cc:460] Webserver started at http://127.3.217.62:34841/ using document root <none> and password file <none>
I20260924 07:10:15.313855 25511 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:15.313925 25511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:15.314114 25511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:15.317075 25511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "94d2281266d1403cbd715fa2d2acb824"
format_stamp: "Formatted at 2026-09-24 07:10:15 on dist-test-slave-2p7j"
server_key: "df0617a7147f6e31c3bb76cfcaca82b9"
server_key_iv: "70a819db2224bf8f6590103152ce275b"
server_key_version: "encryptionkey@0"
I20260924 07:10:15.317832 25511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "94d2281266d1403cbd715fa2d2acb824"
format_stamp: "Formatted at 2026-09-24 07:10:15 on dist-test-slave-2p7j"
server_key: "df0617a7147f6e31c3bb76cfcaca82b9"
server_key_iv: "70a819db2224bf8f6590103152ce275b"
server_key_version: "encryptionkey@0"
I20260924 07:10:15.322570 25511 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:10:15.325505 25526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:15.326853 25511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:10:15.326969 25511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "94d2281266d1403cbd715fa2d2acb824"
format_stamp: "Formatted at 2026-09-24 07:10:15 on dist-test-slave-2p7j"
server_key: "df0617a7147f6e31c3bb76cfcaca82b9"
server_key_iv: "70a819db2224bf8f6590103152ce275b"
server_key_version: "encryptionkey@0"
I20260924 07:10:15.327095 25511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:15.352631 25511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:15.355988 25511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:15.356156 25511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:15.363879 25511 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:37315
I20260924 07:10:15.363929 25578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:37315 every 8 connection(s)
I20260924 07:10:15.364800 25511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:15.367523 25579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:15.368688 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 25511
I20260924 07:10:15.368821 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:10:15.369063 3940 external_mini_cluster.cc:1428] Setting key f52c3d8d3e55441be9915ce5e0e0a893
I20260924 07:10:15.376479 25579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94d2281266d1403cbd715fa2d2acb824: Bootstrap starting.
I20260924 07:10:15.378976 25579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94d2281266d1403cbd715fa2d2acb824: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:15.379732 25579 log.cc:824] T 00000000000000000000000000000000 P 94d2281266d1403cbd715fa2d2acb824: Log is configured to *not* fsync() on all Append() calls
Sep 24 07:10:15 dist-test-slave-2p7j krb5kdc[25498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:10:15.382089 25579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94d2281266d1403cbd715fa2d2acb824: No bootstrap required, opened a new log
I20260924 07:10:15.386057 25579 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94d2281266d1403cbd715fa2d2acb824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d2281266d1403cbd715fa2d2acb824" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37315 } }
I20260924 07:10:15.386286 25579 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94d2281266d1403cbd715fa2d2acb824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:15.386345 25579 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94d2281266d1403cbd715fa2d2acb824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d2281266d1403cbd715fa2d2acb824, State: Initialized, Role: FOLLOWER
I20260924 07:10:15.386734 25579 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94d2281266d1403cbd715fa2d2acb824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d2281266d1403cbd715fa2d2acb824" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37315 } }
I20260924 07:10:15.386864 25579 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94d2281266d1403cbd715fa2d2acb824 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:15.386881 25582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:15.370593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33121 (local address 127.3.217.62:37315)
0924 07:10:15.371241 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:15.371262 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:15.371304 (+ 42us) server_negotiation.cc:407] Connection header received
0924 07:10:15.372132 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:15.372156 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:15.372573 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:15.372936 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:15.374242 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:15.375789 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:15.377049 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:15.377738 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:15.377917 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:15.381460 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:15.381468 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:15.381489 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:15.381534 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:15.383493 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:15.384248 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:15.384255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:15.384260 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:15.384394 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:15.384847 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:15.384850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:15.384851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:15.385220 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:15.385391 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:15.385747 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 07:10:15.386004 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":118,"threads_started":1}
I20260924 07:10:15.386927 25579 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94d2281266d1403cbd715fa2d2acb824 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:15.387020 25579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94d2281266d1403cbd715fa2d2acb824 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:15.388130 25579 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94d2281266d1403cbd715fa2d2acb824 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d2281266d1403cbd715fa2d2acb824" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37315 } }
I20260924 07:10:15.388430 25579 leader_election.cc:302] T 00000000000000000000000000000000 P 94d2281266d1403cbd715fa2d2acb824 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94d2281266d1403cbd715fa2d2acb824; no voters:
I20260924 07:10:15.388691 25579 leader_election.cc:288] T 00000000000000000000000000000000 P 94d2281266d1403cbd715fa2d2acb824 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:10:15.388789 25584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94d2281266d1403cbd715fa2d2acb824 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:15.388999 25584 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94d2281266d1403cbd715fa2d2acb824 [term 1 LEADER]: Becoming Leader. State: Replica: 94d2281266d1403cbd715fa2d2acb824, State: Running, Role: LEADER
I20260924 07:10:15.389312 25584 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94d2281266d1403cbd715fa2d2acb824 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d2281266d1403cbd715fa2d2acb824" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37315 } }
I20260924 07:10:15.389734 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:10:15.389947 25579 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94d2281266d1403cbd715fa2d2acb824 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:15.390744 25585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d2281266d1403cbd715fa2d2acb824 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94d2281266d1403cbd715fa2d2acb824" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d2281266d1403cbd715fa2d2acb824" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37315 } } }
I20260924 07:10:15.390858 25585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d2281266d1403cbd715fa2d2acb824 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:15.391146 25586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d2281266d1403cbd715fa2d2acb824 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94d2281266d1403cbd715fa2d2acb824. Latest consensus state: current_term: 1 leader_uuid: "94d2281266d1403cbd715fa2d2acb824" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d2281266d1403cbd715fa2d2acb824" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 37315 } } }
I20260924 07:10:15.391229 25586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d2281266d1403cbd715fa2d2acb824 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:15.392158 25595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:15.394972 25595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:15.401108 25582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:15.391968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43791 (local address 127.3.217.62:37315)
0924 07:10:15.392072 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:15.392076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:15.392515 (+ 439us) server_negotiation.cc:407] Connection header received
0924 07:10:15.392697 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:15.392699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:15.392738 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:15.392809 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:15.393545 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:15.394291 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:15.395423 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:15.395918 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:15.396029 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:15.397353 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:15.397356 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:15.397359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:15.397383 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:15.398813 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:15.399664 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:15.399668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:15.399670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:15.399769 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:15.400156 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:15.400160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:15.400161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:15.400387 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:15.400545 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:15.400854 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 07:10:15.400955 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:10:15.401582 25595 catalog_manager.cc:1347] Generated new cluster ID: dcdb66dc7a164b7ebf1a11cc06291e2a
I20260924 07:10:15.401712 25595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:15.411654 25595 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:15.412602 25595 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:15.420647 25595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94d2281266d1403cbd715fa2d2acb824: Generated new TSK 0
I20260924 07:10:15.421360 25595 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:10:15.431943 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 25511
W20260924 07:10:15.436683 25496 connection.cc:535] client connection to 127.3.217.62:37315 recv error: Network error: failed to read from TLS socket (remote: 127.3.217.62:37315): Connection reset by peer (error 104)
2026-09-24T07:10:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:10:15 dist-test-slave-2p7j krb5kdc[25610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:10:15 dist-test-slave-2p7j krb5kdc[25610](info): set up 2 sockets
Sep 24 07:10:15 dist-test-slave-2p7j krb5kdc[25610](info): commencing operation
krb5kdc: starting...
W20260924 07:10:16.408126 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:10:16 dist-test-slave-2p7j krb5kdc[25610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:10:16.523957 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:36727
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:45971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:36727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:16.628999 25623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:16.629235 25623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:16.629287 25623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:16.633723 25623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:16.633806 25623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:16.633836 25623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:16.633858 25623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:16.633888 25623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:16.640367 25623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:45971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:36727
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:36727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:10:16.641310 25623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:16.642246 25623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:10:16.651448 25623 server_base.cc:1034] running on GCE node
W20260924 07:10:16.651407 25628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:16.651412 25629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:16.651422 25631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:16.652376 25623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:16.653405 25623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:16.654577 25623 hybrid_clock.cc:648] HybridClock initialized: now 1790233816654560 us; error 28 us; skew 500 ppm
Sep 24 07:10:16 dist-test-slave-2p7j krb5kdc[25610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233816, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:16.657096 25623 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:10:16.657852 25623 webserver.cc:460] Webserver started at http://127.3.217.62:43201/ using document root <none> and password file <none>
I20260924 07:10:16.658334 25623 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:16.658414 25623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:16.658601 25623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:16.660522 25623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "a5e96995eee442a8a988ad7b1089a0ab"
format_stamp: "Formatted at 2026-09-24 07:10:16 on dist-test-slave-2p7j"
server_key: "449d6c15ea22ab277bd031dc9e7e84e0"
server_key_iv: "24879b2f08bdf15b73035649012909e0"
server_key_version: "encryptionkey@0"
I20260924 07:10:16.661032 25623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "a5e96995eee442a8a988ad7b1089a0ab"
format_stamp: "Formatted at 2026-09-24 07:10:16 on dist-test-slave-2p7j"
server_key: "449d6c15ea22ab277bd031dc9e7e84e0"
server_key_iv: "24879b2f08bdf15b73035649012909e0"
server_key_version: "encryptionkey@0"
I20260924 07:10:16.664378 25623 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:10:16.666576 25638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:16.667821 25623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:10:16.667927 25623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "a5e96995eee442a8a988ad7b1089a0ab"
format_stamp: "Formatted at 2026-09-24 07:10:16 on dist-test-slave-2p7j"
server_key: "449d6c15ea22ab277bd031dc9e7e84e0"
server_key_iv: "24879b2f08bdf15b73035649012909e0"
server_key_version: "encryptionkey@0"
I20260924 07:10:16.668008 25623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:16.681298 25623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:16.684195 25623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:16.684355 25623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:16.690636 25623 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:36727
I20260924 07:10:16.690662 25690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:36727 every 8 connection(s)
I20260924 07:10:16.691473 25623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:16.693935 25691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:16.698801 25691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5e96995eee442a8a988ad7b1089a0ab: Bootstrap starting.
I20260924 07:10:16.699225 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 25623
I20260924 07:10:16.699364 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:10:16.699664 3940 external_mini_cluster.cc:1428] Setting key 6eb7463fc008810d51fa1bf6b454aeca
I20260924 07:10:16.701136 25691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5e96995eee442a8a988ad7b1089a0ab: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:16.701781 25691 log.cc:824] T 00000000000000000000000000000000 P a5e96995eee442a8a988ad7b1089a0ab: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:16.703698 25691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5e96995eee442a8a988ad7b1089a0ab: No bootstrap required, opened a new log
I20260924 07:10:16.707439 25691 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5e96995eee442a8a988ad7b1089a0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e96995eee442a8a988ad7b1089a0ab" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36727 } }
I20260924 07:10:16.707698 25691 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5e96995eee442a8a988ad7b1089a0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:16.707790 25691 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5e96995eee442a8a988ad7b1089a0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e96995eee442a8a988ad7b1089a0ab, State: Initialized, Role: FOLLOWER
I20260924 07:10:16.708197 25691 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5e96995eee442a8a988ad7b1089a0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e96995eee442a8a988ad7b1089a0ab" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36727 } }
I20260924 07:10:16.708335 25691 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5e96995eee442a8a988ad7b1089a0ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:16.708398 25691 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5e96995eee442a8a988ad7b1089a0ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:16.708487 25691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5e96995eee442a8a988ad7b1089a0ab [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:16.708765 25694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:16.700836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41743 (local address 127.3.217.62:36727)
0924 07:10:16.701188 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:16.701202 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:16.701236 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:10:16.701844 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:16.701859 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:16.702169 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:16.702417 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:16.702825 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:16.703794 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:16.704966 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:16.705658 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:16.705905 (+ 247us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:16.706750 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:16.706755 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:16.706778 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:16.707118 (+ 340us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:16.707168 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:16.707191 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:16.707299 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:16.707705 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 07:10:16.707922 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":104,"threads_started":1}
I20260924 07:10:16.709313 25691 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5e96995eee442a8a988ad7b1089a0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e96995eee442a8a988ad7b1089a0ab" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36727 } }
I20260924 07:10:16.709613 25691 leader_election.cc:302] T 00000000000000000000000000000000 P a5e96995eee442a8a988ad7b1089a0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5e96995eee442a8a988ad7b1089a0ab; no voters:
I20260924 07:10:16.709908 25691 leader_election.cc:288] T 00000000000000000000000000000000 P a5e96995eee442a8a988ad7b1089a0ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:10:16.709968 25696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5e96995eee442a8a988ad7b1089a0ab [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:16.710173 25696 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5e96995eee442a8a988ad7b1089a0ab [term 1 LEADER]: Becoming Leader. State: Replica: a5e96995eee442a8a988ad7b1089a0ab, State: Running, Role: LEADER
I20260924 07:10:16.710774 25696 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5e96995eee442a8a988ad7b1089a0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e96995eee442a8a988ad7b1089a0ab" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36727 } }
I20260924 07:10:16.710965 25691 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5e96995eee442a8a988ad7b1089a0ab [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:16.711450 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:10:16.711848 25696 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5e96995eee442a8a988ad7b1089a0ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5e96995eee442a8a988ad7b1089a0ab. Latest consensus state: current_term: 1 leader_uuid: "a5e96995eee442a8a988ad7b1089a0ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e96995eee442a8a988ad7b1089a0ab" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36727 } } }
I20260924 07:10:16.711956 25696 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5e96995eee442a8a988ad7b1089a0ab [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:16.712219 25697 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5e96995eee442a8a988ad7b1089a0ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5e96995eee442a8a988ad7b1089a0ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e96995eee442a8a988ad7b1089a0ab" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 36727 } } }
I20260924 07:10:16.712303 25697 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5e96995eee442a8a988ad7b1089a0ab [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:16.712481 25705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:16.715428 25705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:16.718040 25694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:16.713481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49677 (local address 127.3.217.62:36727)
0924 07:10:16.713583 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:16.713587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:16.714348 (+ 761us) server_negotiation.cc:407] Connection header received
0924 07:10:16.714641 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:16.714643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:16.714678 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:16.714735 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:16.714992 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:16.715577 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:16.716512 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:16.716722 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:16.716804 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:16.717545 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:16.717549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:16.717551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:16.717563 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:16.717588 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:16.717601 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:16.717681 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:16.717887 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 07:10:16.717922 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:10:16.720690 25705 catalog_manager.cc:1347] Generated new cluster ID: f37caa5231bd43d5bc32ff52b2635f37
I20260924 07:10:16.720762 25705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:16.755990 25705 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:16.756908 25705 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:16.768685 25705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5e96995eee442a8a988ad7b1089a0ab: Generated new TSK 0
I20260924 07:10:16.769515 25705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:10:16.775735 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 25623
W20260924 07:10:16.780839 25608 connection.cc:535] client connection to 127.3.217.62:36727 recv error: Network error: failed to read from TLS socket (remote: 127.3.217.62:36727): Connection reset by peer (error 104)
2026-09-24T07:10:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:10:16 dist-test-slave-2p7j krb5kdc[25722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:10:16 dist-test-slave-2p7j krb5kdc[25722](info): set up 2 sockets
Sep 24 07:10:16 dist-test-slave-2p7j krb5kdc[25722](info): commencing operation
krb5kdc: starting...
W20260924 07:10:17.735924 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:10:17 dist-test-slave-2p7j krb5kdc[25722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:10:17.859855 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:44847
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:34211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:44847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:17.967895 25735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:17.968155 25735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:17.968212 25735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:17.972143 25735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:17.972201 25735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:17.972229 25735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:17.972249 25735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:17.972271 25735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:17.977632 25735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:34211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:44847
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:44847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:10:17.978607 25735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:17.979576 25735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:17.987987 25741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:17.988047 25740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:17.988075 25743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:17.988137 25735 server_base.cc:1034] running on GCE node
I20260924 07:10:17.989259 25735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:17.990391 25735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:17.991649 25735 hybrid_clock.cc:648] HybridClock initialized: now 1790233817991586 us; error 67 us; skew 500 ppm
Sep 24 07:10:17 dist-test-slave-2p7j krb5kdc[25722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233817, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:18.002276 25735 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:10:18.003602 25735 webserver.cc:460] Webserver started at http://127.3.217.62:38579/ using document root <none> and password file <none>
I20260924 07:10:18.004158 25735 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:18.004225 25735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:18.004410 25735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:18.006498 25735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "e9455207945d425ab21f0ba7b67210b7"
format_stamp: "Formatted at 2026-09-24 07:10:18 on dist-test-slave-2p7j"
server_key: "6babe3b8aa957825ced2ef5e54e4c8d4"
server_key_iv: "d96f08586b820ab1ecf970dc5b88c3cc"
server_key_version: "encryptionkey@0"
I20260924 07:10:18.007023 25735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "e9455207945d425ab21f0ba7b67210b7"
format_stamp: "Formatted at 2026-09-24 07:10:18 on dist-test-slave-2p7j"
server_key: "6babe3b8aa957825ced2ef5e54e4c8d4"
server_key_iv: "d96f08586b820ab1ecf970dc5b88c3cc"
server_key_version: "encryptionkey@0"
I20260924 07:10:18.010602 25735 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 07:10:18.013317 25750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:18.014851 25735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 07:10:18.014986 25735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "e9455207945d425ab21f0ba7b67210b7"
format_stamp: "Formatted at 2026-09-24 07:10:18 on dist-test-slave-2p7j"
server_key: "6babe3b8aa957825ced2ef5e54e4c8d4"
server_key_iv: "d96f08586b820ab1ecf970dc5b88c3cc"
server_key_version: "encryptionkey@0"
I20260924 07:10:18.015089 25735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:18.019914 25735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:18.023491 25735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:18.023700 25735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:18.030702 25735 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:44847
I20260924 07:10:18.030741 25802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:44847 every 8 connection(s)
I20260924 07:10:18.031688 25735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:18.034353 25803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:18.036221 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 25735
I20260924 07:10:18.036339 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:10:18.036604 3940 external_mini_cluster.cc:1428] Setting key 4181c99280bf520fe4f8c5747ecee2fe
I20260924 07:10:18.041231 25803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9455207945d425ab21f0ba7b67210b7: Bootstrap starting.
I20260924 07:10:18.042991 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:18.038073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51315 (local address 127.3.217.62:44847)
0924 07:10:18.038503 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:18.038526 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:18.038568 (+ 42us) server_negotiation.cc:407] Connection header received
0924 07:10:18.039408 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:18.039430 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:18.039800 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:18.040149 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:18.041313 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:18.041319 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:18.041332 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:18.041344 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:18.041540 (+ 196us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:18.041709 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:18.041869 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:18.041988 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 07:10:18.042132 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":119,"threads_started":1}
I20260924 07:10:18.043989 25803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9455207945d425ab21f0ba7b67210b7: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:18.044695 25803 log.cc:824] T 00000000000000000000000000000000 P e9455207945d425ab21f0ba7b67210b7: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:18.046180 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:10:18.047310 25803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9455207945d425ab21f0ba7b67210b7: No bootstrap required, opened a new log
I20260924 07:10:18.049472 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:18.048045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55599 (local address 127.3.217.62:44847)
0924 07:10:18.048152 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:18.048156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:18.048265 (+ 109us) server_negotiation.cc:407] Connection header received
0924 07:10:18.048328 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:18.048331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:18.048365 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:18.048441 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:18.049037 (+ 596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:18.049040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:18.049043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:18.049054 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:18.049081 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:18.049093 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:18.049265 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:18.049319 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 07:10:18.049357 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:10:18.051772 25803 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9455207945d425ab21f0ba7b67210b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9455207945d425ab21f0ba7b67210b7" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 44847 } }
I20260924 07:10:18.051988 25803 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9455207945d425ab21f0ba7b67210b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:18.052029 25803 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9455207945d425ab21f0ba7b67210b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9455207945d425ab21f0ba7b67210b7, State: Initialized, Role: FOLLOWER
I20260924 07:10:18.052376 25803 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9455207945d425ab21f0ba7b67210b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9455207945d425ab21f0ba7b67210b7" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 44847 } }
I20260924 07:10:18.052486 25803 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9455207945d425ab21f0ba7b67210b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:18.052523 25803 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9455207945d425ab21f0ba7b67210b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:18.052583 25803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9455207945d425ab21f0ba7b67210b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:18.053404 25803 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9455207945d425ab21f0ba7b67210b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9455207945d425ab21f0ba7b67210b7" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 44847 } }
I20260924 07:10:18.053714 25803 leader_election.cc:302] T 00000000000000000000000000000000 P e9455207945d425ab21f0ba7b67210b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9455207945d425ab21f0ba7b67210b7; no voters:
I20260924 07:10:18.053962 25803 leader_election.cc:288] T 00000000000000000000000000000000 P e9455207945d425ab21f0ba7b67210b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:10:18.054062 25813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9455207945d425ab21f0ba7b67210b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:18.054260 25813 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9455207945d425ab21f0ba7b67210b7 [term 1 LEADER]: Becoming Leader. State: Replica: e9455207945d425ab21f0ba7b67210b7, State: Running, Role: LEADER
I20260924 07:10:18.054570 25813 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9455207945d425ab21f0ba7b67210b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9455207945d425ab21f0ba7b67210b7" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 44847 } }
I20260924 07:10:18.055119 25803 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9455207945d425ab21f0ba7b67210b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:18.056137 25815 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9455207945d425ab21f0ba7b67210b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9455207945d425ab21f0ba7b67210b7. Latest consensus state: current_term: 1 leader_uuid: "e9455207945d425ab21f0ba7b67210b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9455207945d425ab21f0ba7b67210b7" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 44847 } } }
I20260924 07:10:18.056285 25815 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9455207945d425ab21f0ba7b67210b7 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:18.056751 25822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:18.057081 25814 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9455207945d425ab21f0ba7b67210b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9455207945d425ab21f0ba7b67210b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9455207945d425ab21f0ba7b67210b7" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 44847 } } }
I20260924 07:10:18.057204 25814 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9455207945d425ab21f0ba7b67210b7 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:18.059760 25822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:18.065316 25822 catalog_manager.cc:1347] Generated new cluster ID: 8a10e6fec80c4788841c41946e6ec6ca
I20260924 07:10:18.065378 25822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:18.080404 25822 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:18.080721 25822 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:18.086117 25822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9455207945d425ab21f0ba7b67210b7: Generated new TSK 0
I20260924 07:10:18.086843 25822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:10:18.094200 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 25735
2026-09-24T07:10:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:10:18 dist-test-slave-2p7j krb5kdc[25834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:10:18 dist-test-slave-2p7j krb5kdc[25834](info): set up 2 sockets
Sep 24 07:10:18 dist-test-slave-2p7j krb5kdc[25834](info): commencing operation
krb5kdc: starting...
W20260924 07:10:19.055985 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:10:19 dist-test-slave-2p7j krb5kdc[25834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:10:19.174276 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:45359
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:43525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:45359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:19.275415 25847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:19.275678 25847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:19.275735 25847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:19.279593 25847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:19.279647 25847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:19.279673 25847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:19.279695 25847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:19.279714 25847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:19.284714 25847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:43525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:45359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:45359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:10:19.285615 25847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:19.286513 25847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:10:19.294422 25847 server_base.cc:1034] running on GCE node
W20260924 07:10:19.294452 25852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:19.294443 25855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:19.294421 25853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:19.295113 25847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:19.296139 25847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:19.297453 25847 hybrid_clock.cc:648] HybridClock initialized: now 1790233819297415 us; error 36 us; skew 500 ppm
Sep 24 07:10:19 dist-test-slave-2p7j krb5kdc[25834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233819, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:19.301676 25847 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:10:19.302860 25847 webserver.cc:460] Webserver started at http://127.3.217.62:40607/ using document root <none> and password file <none>
I20260924 07:10:19.303601 25847 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:19.303682 25847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:19.303860 25847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:19.305771 25847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "f5416609c68c4acc8b3c84029e589efd"
format_stamp: "Formatted at 2026-09-24 07:10:19 on dist-test-slave-2p7j"
server_key: "052271b4cc11bd17d4aa532fd61ea8d4"
server_key_iv: "e903e53d2cca7df9098cb77dd0ec4eef"
server_key_version: "encryptionkey@0"
I20260924 07:10:19.306254 25847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "f5416609c68c4acc8b3c84029e589efd"
format_stamp: "Formatted at 2026-09-24 07:10:19 on dist-test-slave-2p7j"
server_key: "052271b4cc11bd17d4aa532fd61ea8d4"
server_key_iv: "e903e53d2cca7df9098cb77dd0ec4eef"
server_key_version: "encryptionkey@0"
I20260924 07:10:19.309906 25847 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:10:19.312201 25862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:19.313606 25847 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:10:19.313714 25847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "f5416609c68c4acc8b3c84029e589efd"
format_stamp: "Formatted at 2026-09-24 07:10:19 on dist-test-slave-2p7j"
server_key: "052271b4cc11bd17d4aa532fd61ea8d4"
server_key_iv: "e903e53d2cca7df9098cb77dd0ec4eef"
server_key_version: "encryptionkey@0"
I20260924 07:10:19.313844 25847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:19.330956 25847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:19.334023 25847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:19.334168 25847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:19.340898 25847 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:45359
I20260924 07:10:19.340943 25914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:45359 every 8 connection(s)
I20260924 07:10:19.341753 25847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:19.344506 25915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:19.349539 25915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5416609c68c4acc8b3c84029e589efd: Bootstrap starting.
I20260924 07:10:19.349632 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 25847
I20260924 07:10:19.349756 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:10:19.350252 3940 external_mini_cluster.cc:1428] Setting key 2f085b9ee63b973dfe807905fc3482fe
I20260924 07:10:19.352018 25915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5416609c68c4acc8b3c84029e589efd: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:19.353451 25915 log.cc:824] T 00000000000000000000000000000000 P f5416609c68c4acc8b3c84029e589efd: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:19.355988 25915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5416609c68c4acc8b3c84029e589efd: No bootstrap required, opened a new log
I20260924 07:10:19.359722 25915 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5416609c68c4acc8b3c84029e589efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5416609c68c4acc8b3c84029e589efd" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45359 } }
I20260924 07:10:19.359932 25915 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5416609c68c4acc8b3c84029e589efd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Sep 24 07:10:19 dist-test-slave-2p7j krb5kdc[25834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:10:19.359984 25915 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5416609c68c4acc8b3c84029e589efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5416609c68c4acc8b3c84029e589efd, State: Initialized, Role: FOLLOWER
I20260924 07:10:19.360365 25915 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5416609c68c4acc8b3c84029e589efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5416609c68c4acc8b3c84029e589efd" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45359 } }
I20260924 07:10:19.360507 25915 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5416609c68c4acc8b3c84029e589efd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:19.360572 25915 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5416609c68c4acc8b3c84029e589efd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:19.360668 25915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5416609c68c4acc8b3c84029e589efd [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:19.361652 25915 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5416609c68c4acc8b3c84029e589efd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5416609c68c4acc8b3c84029e589efd" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45359 } }
I20260924 07:10:19.362004 25915 leader_election.cc:302] T 00000000000000000000000000000000 P f5416609c68c4acc8b3c84029e589efd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5416609c68c4acc8b3c84029e589efd; no voters:
I20260924 07:10:19.362339 25915 leader_election.cc:288] T 00000000000000000000000000000000 P f5416609c68c4acc8b3c84029e589efd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:10:19.362461 25920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5416609c68c4acc8b3c84029e589efd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:19.362648 25920 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5416609c68c4acc8b3c84029e589efd [term 1 LEADER]: Becoming Leader. State: Replica: f5416609c68c4acc8b3c84029e589efd, State: Running, Role: LEADER
I20260924 07:10:19.363001 25920 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5416609c68c4acc8b3c84029e589efd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5416609c68c4acc8b3c84029e589efd" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45359 } }
I20260924 07:10:19.363600 25915 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5416609c68c4acc8b3c84029e589efd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:19.364600 25921 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5416609c68c4acc8b3c84029e589efd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5416609c68c4acc8b3c84029e589efd. Latest consensus state: current_term: 1 leader_uuid: "f5416609c68c4acc8b3c84029e589efd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5416609c68c4acc8b3c84029e589efd" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45359 } } }
I20260924 07:10:19.364722 25921 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5416609c68c4acc8b3c84029e589efd [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:19.365077 25922 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5416609c68c4acc8b3c84029e589efd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5416609c68c4acc8b3c84029e589efd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5416609c68c4acc8b3c84029e589efd" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 45359 } } }
I20260924 07:10:19.365216 25922 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5416609c68c4acc8b3c84029e589efd [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:19.366215 25918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:19.351604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37289 (local address 127.3.217.62:45359)
0924 07:10:19.352021 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:19.352039 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:19.352196 (+ 157us) server_negotiation.cc:407] Connection header received
0924 07:10:19.353476 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:19.353507 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:19.354080 (+ 573us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:19.354465 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:19.355201 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:19.356586 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:19.357426 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:19.358085 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:19.358236 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:19.360500 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:19.360506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:19.360522 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:19.360552 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:19.362575 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:19.363130 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:19.363136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:19.363141 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:19.363271 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:19.363579 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:19.363582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:19.363584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:19.364063 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:19.364239 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:19.365053 (+ 814us) server_negotiation.cc:299] Negotiation successful
0924 07:10:19.365333 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":111,"threads_started":1}
W20260924 07:10:19.368072 25935 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f5416609c68c4acc8b3c84029e589efd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:10:19.368187 25935 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:10:19.368264 25929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:19.368944 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:10:19.369102 25929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:19.374578 25929 catalog_manager.cc:1347] Generated new cluster ID: f340e6cb904447c0bd8eb98183dca5f4
I20260924 07:10:19.374634 25929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:19.379005 25918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:19.370633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50083 (local address 127.3.217.62:45359)
0924 07:10:19.371647 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:19.371650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:19.371860 (+ 210us) server_negotiation.cc:407] Connection header received
0924 07:10:19.371960 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:19.371963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:19.372008 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:19.372076 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:19.372704 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:19.373278 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:19.374047 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:19.374301 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:19.374392 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:19.375948 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:19.375952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:19.375954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:19.375978 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:19.377561 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:19.377950 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:19.377954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:19.377955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:19.378034 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:19.378260 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:19.378263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:19.378265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:19.378466 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:19.378597 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:19.378761 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 07:10:19.378865 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":944}
I20260924 07:10:19.385085 25929 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:19.386416 25929 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:19.392052 25929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5416609c68c4acc8b3c84029e589efd: Generated new TSK 0
I20260924 07:10:19.392702 25929 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:10:19.397569 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 25847
2026-09-24T07:10:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:10:19 dist-test-slave-2p7j krb5kdc[25946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:10:19 dist-test-slave-2p7j krb5kdc[25946](info): set up 2 sockets
Sep 24 07:10:19 dist-test-slave-2p7j krb5kdc[25946](info): commencing operation
krb5kdc: starting...
W20260924 07:10:20.355834 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:10:20 dist-test-slave-2p7j krb5kdc[25946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:10:20.475217 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:33091
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:35799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:33091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:20.580356 25959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:20.580585 25959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:20.580624 25959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:20.584591 25959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:20.584653 25959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:20.584672 25959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:20.584687 25959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:20.584702 25959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:20.589956 25959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:35799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:33091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:33091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:10:20.590888 25959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:20.591856 25959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:20.600049 25965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:20.600097 25967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:20.600073 25959 server_base.cc:1034] running on GCE node
W20260924 07:10:20.600131 25964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:20.600721 25959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:20.601692 25959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:20.602869 25959 hybrid_clock.cc:648] HybridClock initialized: now 1790233820602847 us; error 31 us; skew 500 ppm
Sep 24 07:10:20 dist-test-slave-2p7j krb5kdc[25946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233820, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:20.605432 25959 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:10:20.606230 25959 webserver.cc:460] Webserver started at http://127.3.217.62:41907/ using document root <none> and password file <none>
I20260924 07:10:20.606709 25959 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:20.606765 25959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:20.606911 25959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:20.608785 25959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "e8f51c170892465a8f42884d8013fa94"
format_stamp: "Formatted at 2026-09-24 07:10:20 on dist-test-slave-2p7j"
server_key: "38c09a80db37facc1e1cfda88c5e5fb1"
server_key_iv: "863b5192757faf0bfc29f366486c924d"
server_key_version: "encryptionkey@0"
I20260924 07:10:20.609259 25959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "e8f51c170892465a8f42884d8013fa94"
format_stamp: "Formatted at 2026-09-24 07:10:20 on dist-test-slave-2p7j"
server_key: "38c09a80db37facc1e1cfda88c5e5fb1"
server_key_iv: "863b5192757faf0bfc29f366486c924d"
server_key_version: "encryptionkey@0"
I20260924 07:10:20.613154 25959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 07:10:20.615332 25974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:20.616633 25959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:10:20.616879 25959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "e8f51c170892465a8f42884d8013fa94"
format_stamp: "Formatted at 2026-09-24 07:10:20 on dist-test-slave-2p7j"
server_key: "38c09a80db37facc1e1cfda88c5e5fb1"
server_key_iv: "863b5192757faf0bfc29f366486c924d"
server_key_version: "encryptionkey@0"
I20260924 07:10:20.617090 25959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:20.630263 25959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:20.633142 25959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:20.633293 25959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:20.641644 25959 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:33091
I20260924 07:10:20.641645 26026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:33091 every 8 connection(s)
I20260924 07:10:20.642675 25959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:20.645932 26027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:20.650911 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 25959
I20260924 07:10:20.651031 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:10:20.651326 3940 external_mini_cluster.cc:1428] Setting key 12eab0aaf11dd0e63436d782a674759b
I20260924 07:10:20.651427 26027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f51c170892465a8f42884d8013fa94: Bootstrap starting.
I20260924 07:10:20.653946 26027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8f51c170892465a8f42884d8013fa94: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:20.654626 26027 log.cc:824] T 00000000000000000000000000000000 P e8f51c170892465a8f42884d8013fa94: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:20.656394 26027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f51c170892465a8f42884d8013fa94: No bootstrap required, opened a new log
I20260924 07:10:20.659766 26030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:20.652806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50869 (local address 127.3.217.62:33091)
0924 07:10:20.653244 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:20.653259 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:20.653299 (+ 40us) server_negotiation.cc:407] Connection header received
0924 07:10:20.653900 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:20.653914 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:20.654256 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:20.654544 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:20.654924 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:20.656052 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:20.656909 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:20.657349 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:20.657506 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:20.658246 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:20.658251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:20.658269 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:20.658608 (+ 339us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:20.658655 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:20.658677 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:20.658757 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:20.658894 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 07:10:20.659068 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":127,"threads_started":1}
I20260924 07:10:20.659770 26027 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8f51c170892465a8f42884d8013fa94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f51c170892465a8f42884d8013fa94" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33091 } }
I20260924 07:10:20.660037 26027 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8f51c170892465a8f42884d8013fa94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:20.660106 26027 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8f51c170892465a8f42884d8013fa94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f51c170892465a8f42884d8013fa94, State: Initialized, Role: FOLLOWER
I20260924 07:10:20.660559 26027 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8f51c170892465a8f42884d8013fa94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f51c170892465a8f42884d8013fa94" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33091 } }
I20260924 07:10:20.660709 26027 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8f51c170892465a8f42884d8013fa94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:20.660771 26027 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8f51c170892465a8f42884d8013fa94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:20.660857 26027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8f51c170892465a8f42884d8013fa94 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:20.661731 26027 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8f51c170892465a8f42884d8013fa94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f51c170892465a8f42884d8013fa94" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33091 } }
I20260924 07:10:20.662042 26027 leader_election.cc:302] T 00000000000000000000000000000000 P e8f51c170892465a8f42884d8013fa94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8f51c170892465a8f42884d8013fa94; no voters:
I20260924 07:10:20.662317 26027 leader_election.cc:288] T 00000000000000000000000000000000 P e8f51c170892465a8f42884d8013fa94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:10:20.662395 26032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8f51c170892465a8f42884d8013fa94 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:20.662575 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:10:20.662575 26032 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8f51c170892465a8f42884d8013fa94 [term 1 LEADER]: Becoming Leader. State: Replica: e8f51c170892465a8f42884d8013fa94, State: Running, Role: LEADER
I20260924 07:10:20.662854 26032 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8f51c170892465a8f42884d8013fa94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f51c170892465a8f42884d8013fa94" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33091 } }
I20260924 07:10:20.663491 26027 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8f51c170892465a8f42884d8013fa94 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:20.664517 26033 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f51c170892465a8f42884d8013fa94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8f51c170892465a8f42884d8013fa94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f51c170892465a8f42884d8013fa94" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33091 } } }
I20260924 07:10:20.664664 26033 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f51c170892465a8f42884d8013fa94 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:20.664911 26034 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f51c170892465a8f42884d8013fa94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8f51c170892465a8f42884d8013fa94. Latest consensus state: current_term: 1 leader_uuid: "e8f51c170892465a8f42884d8013fa94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f51c170892465a8f42884d8013fa94" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 33091 } } }
I20260924 07:10:20.665005 26034 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f51c170892465a8f42884d8013fa94 [sys.catalog]: This master's current role is: LEADER
W20260924 07:10:20.668496 26052 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e8f51c170892465a8f42884d8013fa94: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:10:20.668578 26052 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:10:20.668654 26045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:20.669487 26045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:20.670485 26030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:20.665243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40983 (local address 127.3.217.62:33091)
0924 07:10:20.665349 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:20.665353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:20.665544 (+ 191us) server_negotiation.cc:407] Connection header received
0924 07:10:20.665817 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:20.665820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:20.665874 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:20.665924 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:20.666365 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:20.667111 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:20.668482 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:20.668802 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:20.669242 (+ 440us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:20.669799 (+ 557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:20.669803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:20.669805 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:20.669817 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:20.669844 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:20.669857 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:20.670256 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:20.670310 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 07:10:20.670364 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:10:20.675195 26045 catalog_manager.cc:1347] Generated new cluster ID: 7c923787a8ae4bfcb69e5e366509d3fd
I20260924 07:10:20.675264 26045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:20.706856 26045 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:20.707787 26045 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:20.713287 26045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8f51c170892465a8f42884d8013fa94: Generated new TSK 0
I20260924 07:10:20.713865 26045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:10:20.725751 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 25959
2026-09-24T07:10:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:10:20 dist-test-slave-2p7j krb5kdc[26058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:10:20 dist-test-slave-2p7j krb5kdc[26058](info): set up 2 sockets
Sep 24 07:10:20 dist-test-slave-2p7j krb5kdc[26058](info): commencing operation
krb5kdc: starting...
W20260924 07:10:21.676239 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:10:21 dist-test-slave-2p7j krb5kdc[26058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:10:21.791425 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:41991
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:39033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:41991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:21.893481 26071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:21.893707 26071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:21.893760 26071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:21.897660 26071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:21.897712 26071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:21.897738 26071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:21.897760 26071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:21.897789 26071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:21.902760 26071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:39033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:41991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:41991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:10:21.903719 26071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:21.904646 26071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:21.912650 26076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:21.912729 26077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:21.912833 26071 server_base.cc:1034] running on GCE node
W20260924 07:10:21.912729 26079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:21.913740 26071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:21.914976 26071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:21.916152 26071 hybrid_clock.cc:648] HybridClock initialized: now 1790233821916133 us; error 31 us; skew 500 ppm
Sep 24 07:10:21 dist-test-slave-2p7j krb5kdc[26058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233821, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:21.918674 26071 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:10:21.919559 26071 webserver.cc:460] Webserver started at http://127.3.217.62:45615/ using document root <none> and password file <none>
I20260924 07:10:21.920070 26071 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:21.920133 26071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:21.920336 26071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:21.922195 26071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "627566658a4d46f0b7d3969cc8ea5b1a"
format_stamp: "Formatted at 2026-09-24 07:10:21 on dist-test-slave-2p7j"
server_key: "893ba2e1aea0b82e12b6fc5272b6fda3"
server_key_iv: "1ef76f1a05227b3e2b1b44614587c98f"
server_key_version: "encryptionkey@0"
I20260924 07:10:21.922711 26071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "627566658a4d46f0b7d3969cc8ea5b1a"
format_stamp: "Formatted at 2026-09-24 07:10:21 on dist-test-slave-2p7j"
server_key: "893ba2e1aea0b82e12b6fc5272b6fda3"
server_key_iv: "1ef76f1a05227b3e2b1b44614587c98f"
server_key_version: "encryptionkey@0"
I20260924 07:10:21.926198 26071 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 07:10:21.928527 26086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:21.929607 26071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:10:21.929713 26071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "627566658a4d46f0b7d3969cc8ea5b1a"
format_stamp: "Formatted at 2026-09-24 07:10:21 on dist-test-slave-2p7j"
server_key: "893ba2e1aea0b82e12b6fc5272b6fda3"
server_key_iv: "1ef76f1a05227b3e2b1b44614587c98f"
server_key_version: "encryptionkey@0"
I20260924 07:10:21.929806 26071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:21.934300 26071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:21.937687 26071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:21.937857 26071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:21.944850 26071 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:41991
I20260924 07:10:21.944861 26138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:41991 every 8 connection(s)
I20260924 07:10:21.945696 26071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:21.946682 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 26071
I20260924 07:10:21.946851 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:10:21.947149 3940 external_mini_cluster.cc:1428] Setting key a31188cb848a9204389cd678589cd789
I20260924 07:10:21.948760 26139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:21.951653 26142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:21.948344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38977 (local address 127.3.217.62:41991)
0924 07:10:21.948719 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:21.948737 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:21.948771 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:10:21.949383 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:21.949398 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:21.949657 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:21.949908 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:21.950581 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:21.950585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:21.950594 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:21.950603 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:21.950646 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:21.950663 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:21.950761 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:21.950848 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 07:10:21.950964 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":112,"threads_started":1}
I20260924 07:10:21.954107 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:10:21.954262 26139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 627566658a4d46f0b7d3969cc8ea5b1a: Bootstrap starting.
I20260924 07:10:21.956679 26139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 627566658a4d46f0b7d3969cc8ea5b1a: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:21.957470 26139 log.cc:824] T 00000000000000000000000000000000 P 627566658a4d46f0b7d3969cc8ea5b1a: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:21.958078 26142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:21.956218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56477 (local address 127.3.217.62:41991)
0924 07:10:21.956318 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:21.956322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:21.956585 (+ 263us) server_negotiation.cc:407] Connection header received
0924 07:10:21.956662 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:21.956664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:21.956716 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:21.956814 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:21.957484 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:21.957488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:21.957490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:21.957504 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:21.957541 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:21.957556 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:21.957678 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:21.957814 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 07:10:21.957886 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:10:21.959571 26139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 627566658a4d46f0b7d3969cc8ea5b1a: No bootstrap required, opened a new log
I20260924 07:10:21.962822 26139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 627566658a4d46f0b7d3969cc8ea5b1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627566658a4d46f0b7d3969cc8ea5b1a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 41991 } }
I20260924 07:10:21.963017 26139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 627566658a4d46f0b7d3969cc8ea5b1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:21.963052 26139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 627566658a4d46f0b7d3969cc8ea5b1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 627566658a4d46f0b7d3969cc8ea5b1a, State: Initialized, Role: FOLLOWER
I20260924 07:10:21.963420 26139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 627566658a4d46f0b7d3969cc8ea5b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627566658a4d46f0b7d3969cc8ea5b1a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 41991 } }
I20260924 07:10:21.963577 26139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 627566658a4d46f0b7d3969cc8ea5b1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:21.963642 26139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 627566658a4d46f0b7d3969cc8ea5b1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:21.963733 26139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 627566658a4d46f0b7d3969cc8ea5b1a [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:21.964556 26139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 627566658a4d46f0b7d3969cc8ea5b1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627566658a4d46f0b7d3969cc8ea5b1a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 41991 } }
I20260924 07:10:21.964852 26139 leader_election.cc:302] T 00000000000000000000000000000000 P 627566658a4d46f0b7d3969cc8ea5b1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 627566658a4d46f0b7d3969cc8ea5b1a; no voters:
I20260924 07:10:21.965101 26139 leader_election.cc:288] T 00000000000000000000000000000000 P 627566658a4d46f0b7d3969cc8ea5b1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:10:21.965242 26149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 627566658a4d46f0b7d3969cc8ea5b1a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:21.965444 26149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 627566658a4d46f0b7d3969cc8ea5b1a [term 1 LEADER]: Becoming Leader. State: Replica: 627566658a4d46f0b7d3969cc8ea5b1a, State: Running, Role: LEADER
I20260924 07:10:21.965709 26149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 627566658a4d46f0b7d3969cc8ea5b1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627566658a4d46f0b7d3969cc8ea5b1a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 41991 } }
I20260924 07:10:21.966172 26139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 627566658a4d46f0b7d3969cc8ea5b1a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:21.967372 26150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 627566658a4d46f0b7d3969cc8ea5b1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "627566658a4d46f0b7d3969cc8ea5b1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627566658a4d46f0b7d3969cc8ea5b1a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 41991 } } }
I20260924 07:10:21.967476 26150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 627566658a4d46f0b7d3969cc8ea5b1a [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:21.967804 26151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 627566658a4d46f0b7d3969cc8ea5b1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 627566658a4d46f0b7d3969cc8ea5b1a. Latest consensus state: current_term: 1 leader_uuid: "627566658a4d46f0b7d3969cc8ea5b1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627566658a4d46f0b7d3969cc8ea5b1a" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 41991 } } }
I20260924 07:10:21.967881 26151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 627566658a4d46f0b7d3969cc8ea5b1a [sys.catalog]: This master's current role is: LEADER
W20260924 07:10:21.970589 26164 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 627566658a4d46f0b7d3969cc8ea5b1a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:10:21.970652 26164 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:10:21.970722 26159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:21.971356 26159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:21.975876 26159 catalog_manager.cc:1347] Generated new cluster ID: 9452478009824909a8505324711a6def
I20260924 07:10:21.975935 26159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:21.995214 26159 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:21.995492 26159 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:22.007179 26159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 627566658a4d46f0b7d3969cc8ea5b1a: Generated new TSK 0
I20260924 07:10:22.007818 26159 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:10:22.011667 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 26071
2026-09-24T07:10:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1285 ms)
[----------] 18 tests from AuthTokenIssuingTest (24390 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:10:22 dist-test-slave-2p7j krb5kdc[26170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:10:22 dist-test-slave-2p7j krb5kdc[26170](info): set up 2 sockets
Sep 24 07:10:22 dist-test-slave-2p7j krb5kdc[26170](info): commencing operation
krb5kdc: starting...
W20260924 07:10:22.976060 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:10:23 dist-test-slave-2p7j krb5kdc[26170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:10:23.094750 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:42411
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:45447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:42411,127.3.217.61:39877,127.3.217.60:34077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:23.203485 26183 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:10:23.203802 26183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:23.203852 26183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:23.203928 26183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:23.207967 26183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:23.208025 26183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:23.208056 26183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:23.208089 26183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:23.208122 26183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:23.213294 26183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:45447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:42411,127.3.217.61:39877,127.3.217.60:34077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:42411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:10:23.214286 26183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:23.215261 26183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:23.223769 26191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:23.223728 26188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:23.223802 26183 server_base.cc:1034] running on GCE node
W20260924 07:10:23.223721 26189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:23.224561 26183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:23.225660 26183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:23.226868 26183 hybrid_clock.cc:648] HybridClock initialized: now 1790233823226819 us; error 62 us; skew 500 ppm
Sep 24 07:10:23 dist-test-slave-2p7j krb5kdc[26170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:23.235352 26183 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:10:23.236330 26183 webserver.cc:460] Webserver started at http://127.3.217.62:38643/ using document root <none> and password file <none>
I20260924 07:10:23.236871 26183 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:23.236936 26183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:23.237120 26183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:23.239290 26183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "5d08fa743f614c8f8c7e3f500a89e7bb"
format_stamp: "Formatted at 2026-09-24 07:10:23 on dist-test-slave-2p7j"
server_key: "ef2b395c5fddaf7a3f9cabb02f99a440"
server_key_iv: "e4a308954b13f774f15c5c27c97ac631"
server_key_version: "encryptionkey@0"
I20260924 07:10:23.239930 26183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "5d08fa743f614c8f8c7e3f500a89e7bb"
format_stamp: "Formatted at 2026-09-24 07:10:23 on dist-test-slave-2p7j"
server_key: "ef2b395c5fddaf7a3f9cabb02f99a440"
server_key_iv: "e4a308954b13f774f15c5c27c97ac631"
server_key_version: "encryptionkey@0"
I20260924 07:10:23.243736 26183 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:10:23.246520 26198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:23.247885 26183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:10:23.247995 26183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "5d08fa743f614c8f8c7e3f500a89e7bb"
format_stamp: "Formatted at 2026-09-24 07:10:23 on dist-test-slave-2p7j"
server_key: "ef2b395c5fddaf7a3f9cabb02f99a440"
server_key_iv: "e4a308954b13f774f15c5c27c97ac631"
server_key_version: "encryptionkey@0"
I20260924 07:10:23.248102 26183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:23.270874 26183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:23.274521 26183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:23.274681 26183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:23.281221 26183 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:42411
I20260924 07:10:23.281456 26250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:42411 every 8 connection(s)
I20260924 07:10:23.282941 26183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:23.285681 26251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:23.289245 26251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42411 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:10:23.291121 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 26183
I20260924 07:10:23.291242 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:10:23.291584 3940 external_mini_cluster.cc:1428] Setting key c501137675f7855015b6819a05b38e6a
WARNING: no policy specified for kudu/127.3.217.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.61@KRBTEST.COM" created.
Sep 24 07:10:23 dist-test-slave-2p7j krb5kdc[26170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:10:23.308496 26254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.291362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42411 (local address 127.0.0.1:38484)
0924 07:10:23.291990 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.292026 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:23.292819 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:23.293550 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:23.293556 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:23.294033 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:23.294619 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:23.294657 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.296178 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.296181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:23.297316 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:23.297328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.297981 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.297985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:23.298107 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.298790 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:23.298811 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:23.302676 (+ 3865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:23.304827 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:23.304835 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:23.304846 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:23.305048 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:23.305734 (+ 686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:23.305737 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:23.305740 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:23.305880 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:23.306673 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:23.306680 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:23.306843 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 07:10:23.307011 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 07:10:23.307482 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":145,"spinlock_wait_cycles":30464,"thread_start_us":171,"threads_started":1}
I20260924 07:10:23.308496 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.291361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38484 (local address 127.3.217.62:42411)
0924 07:10:23.291778 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.291795 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.292122 (+ 327us) server_negotiation.cc:407] Connection header received
0924 07:10:23.293134 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.293138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.293303 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.293446 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.294953 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.296051 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.297456 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.297815 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.297938 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.302860 (+ 4922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:23.302868 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:23.302881 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:23.302906 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:23.304701 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.305238 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.305258 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.305259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.305498 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.305984 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.305987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.305989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.306474 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:23.306703 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:23.307116 (+ 413us) server_negotiation.cc:299] Negotiation successful
0924 07:10:23.307424 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":287,"thread_start_us":143,"threads_started":1}
I20260924 07:10:23.311069 26251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:10:23.312276 26254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.311809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:39877 (local address 127.0.0.1:47024)
0924 07:10:23.311965 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.312173 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.61:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 07:10:23.312654 26201 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.217.61:39877: connect: Connection refused (error 111)
I20260924 07:10:23.314054 26254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.313757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:39877 (local address 127.0.0.1:47030)
0924 07:10:23.313921 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.313973 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.61:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260924 07:10:23.314250 26251 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.61:39877: Network error: Client connection negotiation failed: client connection to 127.3.217.61:39877: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.61.keytab.
Entry for principal kudu/127.3.217.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.61.keytab.
I20260924 07:10:23.319597 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.61:39877
--webserver_interface=127.3.217.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.61.keytab
--principal=kudu/127.3.217.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:45447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:42411,127.3.217.61:39877,127.3.217.60:34077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:10:23.363556 26251 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 } attempt: 1
I20260924 07:10:23.364639 26254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.364315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:39877 (local address 127.0.0.1:47038)
0924 07:10:23.364506 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.364565 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.61:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 07:10:23.365386 26254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.365157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:39877 (local address 127.0.0.1:47046)
0924 07:10:23.365283 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.365330 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.61:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 07:10:23.365515 26251 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.61:39877: Network error: Client connection negotiation failed: client connection to 127.3.217.61:39877: connect: Connection refused (error 111)
W20260924 07:10:23.430330 26258 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:10:23.430598 26258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:23.430620 26258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:23.430665 26258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 07:10:23.433665 26251 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 } attempt: 2
I20260924 07:10:23.434821 26254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.434449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:39877 (local address 127.0.0.1:47054)
0924 07:10:23.434665 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.434743 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.61:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260924 07:10:23.435256 26258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:23.435369 26258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:23.435405 26258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:23.435431 26258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:23.435461 26258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:23.435693 26254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.435423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:39877 (local address 127.0.0.1:47064)
0924 07:10:23.435588 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.435642 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.61:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260924 07:10:23.435997 26251 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.61:39877: Network error: Client connection negotiation failed: client connection to 127.3.217.61:39877: connect: Connection refused (error 111)
I20260924 07:10:23.442791 26258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:45447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:42411,127.3.217.61:39877,127.3.217.60:34077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.61.keytab
--principal=kudu/127.3.217.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.61:39877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.217.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:10:23.443885 26258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:23.444892 26258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:23.453434 26264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:23.453433 26263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:23.453433 26266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:23.453684 26258 server_base.cc:1034] running on GCE node
I20260924 07:10:23.454283 26258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:23.455374 26258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:23.456640 26258 hybrid_clock.cc:648] HybridClock initialized: now 1790233823456597 us; error 40 us; skew 500 ppm
Sep 24 07:10:23 dist-test-slave-2p7j krb5kdc[26170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:23.459545 26258 init.cc:377] Logged in from keytab as kudu/127.3.217.61@KRBTEST.COM (short username kudu)
I20260924 07:10:23.460407 26258 webserver.cc:460] Webserver started at http://127.3.217.61:37711/ using document root <none> and password file <none>
I20260924 07:10:23.460919 26258 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:23.460983 26258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:23.461172 26258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:23.463299 26258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-1/data/instance:
uuid: "d541f35e0ebe47f2b256277d8f6f885e"
format_stamp: "Formatted at 2026-09-24 07:10:23 on dist-test-slave-2p7j"
server_key: "924d7883d026dc815c1c5400f4947293"
server_key_iv: "6de85a8b45ce3328b0f0fd89514d9767"
server_key_version: "encryptionkey@0"
I20260924 07:10:23.463899 26258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-1/wal/instance:
uuid: "d541f35e0ebe47f2b256277d8f6f885e"
format_stamp: "Formatted at 2026-09-24 07:10:23 on dist-test-slave-2p7j"
server_key: "924d7883d026dc815c1c5400f4947293"
server_key_iv: "6de85a8b45ce3328b0f0fd89514d9767"
server_key_version: "encryptionkey@0"
I20260924 07:10:23.468056 26258 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:10:23.470592 26273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:23.472041 26258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:10:23.472167 26258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-1/wal
uuid: "d541f35e0ebe47f2b256277d8f6f885e"
format_stamp: "Formatted at 2026-09-24 07:10:23 on dist-test-slave-2p7j"
server_key: "924d7883d026dc815c1c5400f4947293"
server_key_iv: "6de85a8b45ce3328b0f0fd89514d9767"
server_key_version: "encryptionkey@0"
I20260924 07:10:23.472273 26258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:23.484897 26258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:23.487977 26258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:23.488144 26258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:23.494843 26258 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.61:39877
I20260924 07:10:23.495113 26325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.61:39877 every 8 connection(s)
I20260924 07:10:23.495797 26258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-1/data/info.pb
I20260924 07:10:23.497154 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 26258
I20260924 07:10:23.497424 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-1/wal/instance
I20260924 07:10:23.497843 3940 external_mini_cluster.cc:1428] Setting key b86752a9fa0cf6ab76367e2adebe58b9
I20260924 07:10:23.498692 26326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:23.502282 26326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42411 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.217.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.60@KRBTEST.COM" created.
Sep 24 07:10:23 dist-test-slave-2p7j krb5kdc[26170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.61@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:10:23.515362 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.504105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38498 (local address 127.3.217.62:42411)
0924 07:10:23.504261 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.504266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.505036 (+ 770us) server_negotiation.cc:407] Connection header received
0924 07:10:23.505964 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.505968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.506026 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.506131 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.507433 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.508040 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.509489 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.509811 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.509910 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.512186 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:23.512189 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:23.512191 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:23.512213 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:23.513452 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.513927 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.513931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.513933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.514002 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.514302 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.514305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.514307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.514477 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:23.514557 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:23.515048 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 07:10:23.515171 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 07:10:23.516191 26329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.504399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42411 (local address 127.0.0.1:38498)
0924 07:10:23.504873 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.504924 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:23.505761 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:23.506386 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:23.506392 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:23.506737 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:23.507270 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:23.507290 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.508179 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.508250 (+ 71us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:23.509356 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:23.509366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.509967 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.509970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:23.510118 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.510675 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:23.510691 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:23.512027 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:23.513557 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:23.513563 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:23.513578 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:23.513837 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:23.514079 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:23.514082 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:23.514084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:23.514217 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:23.514571 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:23.514578 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:23.514822 (+ 244us) client_negotiation.cc:769] Sending connection context
0924 07:10:23.514982 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 07:10:23.515303 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":121,"threads_started":1}
I20260924 07:10:23.517544 26326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:10:23 dist-test-slave-2p7j krb5kdc[26170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.61@KRBTEST.COM for kudu/127.3.217.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.60.keytab.
Entry for principal kudu/127.3.217.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.60.keytab.
I20260924 07:10:23.524742 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.60:34077
--webserver_interface=127.3.217.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.60.keytab
--principal=kudu/127.3.217.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:45447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:42411,127.3.217.61:39877,127.3.217.60:34077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:10:23.527240 26251 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 } attempt: 3
I20260924 07:10:23.527822 26329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.517987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:39877 (local address 127.0.0.1:47078)
0924 07:10:23.518074 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.518087 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:23.518155 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:23.518759 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:23.518761 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:23.518884 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:23.518970 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:23.518976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.520101 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.520104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:23.520694 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:23.520703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.521809 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.521814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:23.521929 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.522368 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:23.522382 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:23.523686 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:23.525787 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:23.525790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:23.525792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:23.525967 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:23.526308 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:23.526311 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:23.526312 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:23.526395 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:23.526962 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:23.526966 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:23.527110 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 07:10:23.527294 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 07:10:23.527545 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 07:10:23.527851 26331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.518113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47078 (local address 127.3.217.61:39877)
0924 07:10:23.518445 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.518453 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.518473 (+ 20us) server_negotiation.cc:407] Connection header received
0924 07:10:23.518505 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.518509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.518647 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.518752 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.519084 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.519964 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.520899 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.521616 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.521761 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.523847 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:23.523856 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:23.523866 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:23.523887 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:23.525676 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.526076 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.526083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.526084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.526176 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.526479 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.526482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.526483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.526771 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:23.526910 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:23.527398 (+ 488us) server_negotiation.cc:299] Negotiation successful
0924 07:10:23.527617 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":233,"thread_start_us":134,"threads_started":1}
I20260924 07:10:23.530759 26326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:10:23.531863 26329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.531331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:34077 (local address 127.0.0.1:54446)
0924 07:10:23.531559 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.531785 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:34077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260924 07:10:23.532440 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.217.60:34077: connect: Connection refused (error 111)
I20260924 07:10:23.533666 26329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.533447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:34077 (local address 127.0.0.1:54460)
0924 07:10:23.533584 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.533620 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:34077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260924 07:10:23.533830 26326 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.60:34077: Network error: Client connection negotiation failed: client connection to 127.3.217.60:34077: connect: Connection refused (error 111)
Sep 24 07:10:23 dist-test-slave-2p7j krb5kdc[26170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for kudu/127.3.217.61@KRBTEST.COM
I20260924 07:10:23.537505 26254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.527968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:39877 (local address 127.0.0.1:47092)
0924 07:10:23.528133 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.528151 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:23.528235 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:23.528828 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:23.528831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:23.529008 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:23.529118 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:23.529124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.530186 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.530191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:23.531766 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:23.531779 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.532435 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.532439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:23.532555 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.533022 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:23.533033 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:23.534337 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:23.536177 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:23.536180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:23.536181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:23.536387 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:23.536669 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:23.536672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:23.536673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:23.536777 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:23.537121 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:23.537123 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:23.537203 (+ 80us) client_negotiation.cc:769] Sending connection context
0924 07:10:23.537280 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 07:10:23.537389 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 07:10:23.537595 26333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.528308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47092 (local address 127.3.217.61:39877)
0924 07:10:23.528600 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.528606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.528617 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:10:23.528646 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.528649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.528690 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.528779 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.529231 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.530001 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.531913 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.532297 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.532408 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.534481 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:23.534485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:23.534488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:23.534513 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:23.536069 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.536474 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.536477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.536479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.536556 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.536856 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.536859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.536861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.537047 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:23.537136 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:23.537334 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 07:10:23.537442 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":108,"threads_started":1}
I20260924 07:10:23.538210 26251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:10:23.538806 26254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.538614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:34077 (local address 127.0.0.1:54462)
0924 07:10:23.538704 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.538745 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:34077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 07:10:23.539371 26254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.539160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:34077 (local address 127.0.0.1:54476)
0924 07:10:23.539283 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.539324 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:34077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260924 07:10:23.539474 26251 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.60:34077: Network error: Client connection negotiation failed: client connection to 127.3.217.60:34077: connect: Connection refused (error 111)
I20260924 07:10:23.570672 26251 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } attempt: 1
I20260924 07:10:23.571637 26254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.571311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:34077 (local address 127.0.0.1:54488)
0924 07:10:23.571455 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.571558 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:34077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 07:10:23.572350 26254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.572133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:34077 (local address 127.0.0.1:54502)
0924 07:10:23.572249 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.572297 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:34077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 07:10:23.572482 26251 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.60:34077: Network error: Client connection negotiation failed: client connection to 127.3.217.60:34077: connect: Connection refused (error 111)
I20260924 07:10:23.583086 26326 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } attempt: 1
I20260924 07:10:23.583799 26329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.583576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:34077 (local address 127.0.0.1:54516)
0924 07:10:23.583688 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.583738 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:34077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 07:10:23.584494 26329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.584272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:34077 (local address 127.0.0.1:54528)
0924 07:10:23.584387 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.584433 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:34077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 07:10:23.584611 26326 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.60:34077: Network error: Client connection negotiation failed: client connection to 127.3.217.60:34077: connect: Connection refused (error 111)
W20260924 07:10:23.640651 26332 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:10:23.640928 26332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:23.640950 26332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:23.640995 26332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:23.645241 26332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:23.645310 26332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:23.645331 26332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:23.645346 26332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:23.645360 26332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:23.647644 26251 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } attempt: 2
I20260924 07:10:23.648730 26254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.648349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:34077 (local address 127.0.0.1:54538)
0924 07:10:23.648542 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.648618 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:34077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 07:10:23.649812 26254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.649523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:34077 (local address 127.0.0.1:54544)
0924 07:10:23.649666 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.649726 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:34077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 07:10:23.650029 26251 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.60:34077: Network error: Client connection negotiation failed: client connection to 127.3.217.60:34077: connect: Connection refused (error 111)
I20260924 07:10:23.651535 26332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:45447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:42411,127.3.217.61:39877,127.3.217.60:34077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.60.keytab
--principal=kudu/127.3.217.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.60:34077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.217.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:10:23.652563 26332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:23.652765 26326 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } attempt: 2
I20260924 07:10:23.653486 26329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.653186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:34077 (local address 127.0.0.1:54550)
0924 07:10:23.653351 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.653412 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:34077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 07:10:23.653720 26332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:10:23.654088 26329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.653909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:34077 (local address 127.0.0.1:54552)
0924 07:10:23.654002 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.654045 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:34077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 07:10:23.654191 26326 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.60:34077: Network error: Client connection negotiation failed: client connection to 127.3.217.60:34077: connect: Connection refused (error 111)
W20260924 07:10:23.662170 26342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:23.662174 26340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:23.662564 26339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:23.663545 26332 server_base.cc:1034] running on GCE node
I20260924 07:10:23.664400 26332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:23.665740 26332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:23.666954 26332 hybrid_clock.cc:648] HybridClock initialized: now 1790233823666931 us; error 50 us; skew 500 ppm
Sep 24 07:10:23 dist-test-slave-2p7j krb5kdc[26170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:23.670825 26332 init.cc:377] Logged in from keytab as kudu/127.3.217.60@KRBTEST.COM (short username kudu)
I20260924 07:10:23.671862 26332 webserver.cc:460] Webserver started at http://127.3.217.60:40019/ using document root <none> and password file <none>
I20260924 07:10:23.672412 26332 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:23.672482 26332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:23.672674 26332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:23.674934 26332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-2/data/instance:
uuid: "af842177770d4d77bf63f3abe19aec14"
format_stamp: "Formatted at 2026-09-24 07:10:23 on dist-test-slave-2p7j"
server_key: "874d451df59d23a283429594713579b0"
server_key_iv: "e04c1ba012512ca4e27accbde6d48e45"
server_key_version: "encryptionkey@0"
I20260924 07:10:23.675486 26332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-2/wal/instance:
uuid: "af842177770d4d77bf63f3abe19aec14"
format_stamp: "Formatted at 2026-09-24 07:10:23 on dist-test-slave-2p7j"
server_key: "874d451df59d23a283429594713579b0"
server_key_iv: "e04c1ba012512ca4e27accbde6d48e45"
server_key_version: "encryptionkey@0"
I20260924 07:10:23.679811 26332 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:10:23.683004 26349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:23.684756 26332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:10:23.684909 26332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-2/wal
uuid: "af842177770d4d77bf63f3abe19aec14"
format_stamp: "Formatted at 2026-09-24 07:10:23 on dist-test-slave-2p7j"
server_key: "874d451df59d23a283429594713579b0"
server_key_iv: "e04c1ba012512ca4e27accbde6d48e45"
server_key_version: "encryptionkey@0"
I20260924 07:10:23.685010 26332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:23.705199 26332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:23.708748 26332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:23.708926 26332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:23.716365 26332 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.60:34077
I20260924 07:10:23.716888 26401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.60:34077 every 8 connection(s)
I20260924 07:10:23.717258 26332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-2/data/info.pb
I20260924 07:10:23.719967 26402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:23.721481 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 26332
I20260924 07:10:23.721632 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233541052846-3940-0/minicluster-data/master-2/wal/instance
I20260924 07:10:23.721982 3940 external_mini_cluster.cc:1428] Setting key ad676f37dfb70988a968bfbe5b1f539a
I20260924 07:10:23.724004 26402 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42411 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:10:23 dist-test-slave-2p7j krb5kdc[26170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.60@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
Sep 24 07:10:23 dist-test-slave-2p7j krb5kdc[26170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.60@KRBTEST.COM
I20260924 07:10:23.736951 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.725958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38508 (local address 127.3.217.62:42411)
0924 07:10:23.726100 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.726105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.726636 (+ 531us) server_negotiation.cc:407] Connection header received
0924 07:10:23.726776 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.726779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.726831 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.726916 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.727639 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.728372 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.729852 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.730237 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.730362 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.733240 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:23.733244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:23.733247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:23.733271 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:23.734737 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.735370 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.735373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.735374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.735454 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.735834 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.735838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.735840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.736067 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:23.736152 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:23.736670 (+ 518us) server_negotiation.cc:299] Negotiation successful
0924 07:10:23.736782 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 07:10:23.737784 26405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.723815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54556 (local address 127.3.217.60:34077)
0924 07:10:23.724403 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.724429 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.724492 (+ 63us) server_negotiation.cc:407] Connection header received
0924 07:10:23.725364 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.725385 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.725720 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.726011 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.726853 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.728225 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.729682 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.730405 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.730531 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.733653 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:23.733659 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:23.733667 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:23.733687 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:23.735583 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.735987 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.735990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.735992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.736055 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.736294 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.736296 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.736298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.736541 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:23.736696 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:23.736923 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 07:10:23.737039 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":205,"threads_started":1}
I20260924 07:10:23.737782 26406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.725855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42411 (local address 127.0.0.1:38508)
0924 07:10:23.726520 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.726553 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:23.726667 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:23.726992 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:23.726996 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:23.727296 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:23.727463 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:23.727472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.728588 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.728591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:23.729697 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:23.729710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.730470 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.730476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:23.730656 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.731361 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:23.731380 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:23.733097 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:23.734859 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:23.734868 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:23.734877 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:23.735228 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:23.735588 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:23.735591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:23.735592 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:23.735744 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:23.736158 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:23.736162 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:23.736397 (+ 235us) client_negotiation.cc:769] Sending connection context
0924 07:10:23.736600 (+ 203us) client_negotiation.cc:240] Negotiation successful
0924 07:10:23.736866 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":163,"threads_started":1}
I20260924 07:10:23.739055 26402 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:10:23 dist-test-slave-2p7j krb5kdc[26170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.60@KRBTEST.COM for kudu/127.3.217.61@KRBTEST.COM
I20260924 07:10:23.745332 26326 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } attempt: 3
Sep 24 07:10:23 dist-test-slave-2p7j krb5kdc[26170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:10:23.749239 26251 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } attempt: 3
I20260924 07:10:23.749655 26333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.739781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47102 (local address 127.3.217.61:39877)
0924 07:10:23.739894 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.739899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.739913 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:10:23.739954 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.739957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.740000 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.740069 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.740498 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.741270 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.742315 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.742709 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.742826 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.745021 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:23.745024 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:23.745026 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:23.745046 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:23.747170 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.748448 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.748451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.748453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.748534 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.748812 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.748815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.748817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.749016 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:23.749081 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:23.749409 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 07:10:23.749512 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 07:10:23 dist-test-slave-2p7j krb5kdc[26170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.61@KRBTEST.COM for kudu/127.3.217.60@KRBTEST.COM
I20260924 07:10:23.750394 26406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.739654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:39877 (local address 127.0.0.1:47102)
0924 07:10:23.739793 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.739806 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:23.739876 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:23.740090 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:23.740093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:23.740246 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:23.740350 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:23.740357 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.741371 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.741374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:23.742197 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:23.742205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.742852 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.742855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:23.742930 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.743254 (+ 324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:23.743266 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:23.744807 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:23.747279 (+ 2472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:23.747282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:23.747285 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:23.748361 (+ 1076us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:23.748636 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:23.748639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:23.748641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:23.748718 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:23.749152 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:23.749155 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:23.749233 (+ 78us) client_negotiation.cc:769] Sending connection context
0924 07:10:23.750135 (+ 902us) client_negotiation.cc:240] Negotiation successful
0924 07:10:23.750259 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 07:10:23.750929 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.740402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38516 (local address 127.3.217.62:42411)
0924 07:10:23.740503 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.740507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.740518 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:10:23.740588 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.740590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.740620 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.740668 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.741304 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.742044 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.743054 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.743350 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.743449 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.746374 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:23.746378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:23.746380 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:23.746401 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:23.748049 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.748494 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.748497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.748499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.748567 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.748829 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.748831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.748833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.749003 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:23.749197 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:23.750253 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0924 07:10:23.750779 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:10:23.751219 26402 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:10:23.754601 26329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.745681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:34077 (local address 127.0.0.1:54568)
0924 07:10:23.745818 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.745832 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:23.745899 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:23.746185 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:23.746187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:23.746339 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:23.746437 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:23.746444 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.747310 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.747314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:23.748165 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:23.748175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.748756 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.748760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:23.748844 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.749256 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:23.749269 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:23.750658 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:23.752917 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:23.752920 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:23.752922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:23.753153 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:23.753436 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:23.753439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:23.753441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:23.753518 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:23.753909 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:23.753913 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:23.754071 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 07:10:23.754286 (+ 215us) client_negotiation.cc:240] Negotiation successful
0924 07:10:23.754454 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 07:10:23.755069 26405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.745889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54568 (local address 127.3.217.60:34077)
0924 07:10:23.746006 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.746010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.746023 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:10:23.746059 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.746061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.746099 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.746171 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.746565 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.747184 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.748285 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.748602 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.748738 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.750784 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:23.750787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:23.750790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:23.750811 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:23.752807 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.753255 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.753258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.753260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.753336 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.753610 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.753613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.753615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.753787 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:23.753874 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:23.754833 (+ 959us) server_negotiation.cc:299] Negotiation successful
0924 07:10:23.754933 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 07:10:23 dist-test-slave-2p7j krb5kdc[26170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for kudu/127.3.217.60@KRBTEST.COM
Sep 24 07:10:23 dist-test-slave-2p7j krb5kdc[26170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.60@KRBTEST.COM for kudu/127.3.217.60@KRBTEST.COM
I20260924 07:10:23.758390 26326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d541f35e0ebe47f2b256277d8f6f885e: Bootstrap starting.
Sep 24 07:10:23 dist-test-slave-2p7j krb5kdc[26170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.61@KRBTEST.COM
I20260924 07:10:23.760265 26406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.751708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:34077 (local address 127.0.0.1:54586)
0924 07:10:23.751799 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.751811 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:23.751874 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:23.752301 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:23.752303 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:23.752529 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:23.752609 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:23.752615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.753705 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.753710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:23.754909 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:23.754918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.755412 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.755415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:23.755485 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.755941 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:23.755954 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:23.756984 (+ 1030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:23.758726 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:23.758729 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:23.758731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:23.758931 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:23.759174 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:23.759177 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:23.759179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:23.759250 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:23.759610 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:23.759613 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:23.759689 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 07:10:23.759767 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 07:10:23.760089 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":52}
I20260924 07:10:23.760365 26408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.751877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54586 (local address 127.3.217.60:34077)
0924 07:10:23.752104 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.752116 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.752127 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:10:23.752156 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.752158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.752194 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.752396 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.752791 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.753523 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.755021 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.755306 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.756354 (+ 1048us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.757095 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:23.757097 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:23.757099 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:23.757117 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:23.758373 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.759015 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.759018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.759020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.759088 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.759332 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.759335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.759336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.759529 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:23.759610 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:23.759817 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 07:10:23.760164 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":246,"server-negotiator.queue_time_us":171,"thread_start_us":97,"threads_started":1}
I20260924 07:10:23.761160 26326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d541f35e0ebe47f2b256277d8f6f885e: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:23.762023 26326 log.cc:824] T 00000000000000000000000000000000 P d541f35e0ebe47f2b256277d8f6f885e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:23.762813 26333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.752187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47116 (local address 127.3.217.61:39877)
0924 07:10:23.752273 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.752276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.754022 (+ 1746us) server_negotiation.cc:407] Connection header received
0924 07:10:23.754177 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.754180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.754218 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.754261 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.755024 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.755755 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.756690 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.756967 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.757031 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.759612 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:23.759616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:23.759618 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:23.759643 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:23.761082 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.761540 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.761544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.761545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.761623 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.761913 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.761917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.761918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.762105 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:23.762230 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:23.762592 (+ 362us) server_negotiation.cc:299] Negotiation successful
0924 07:10:23.762688 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 07:10:23.763456 26402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af842177770d4d77bf63f3abe19aec14: Bootstrap starting.
I20260924 07:10:23.765571 26326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d541f35e0ebe47f2b256277d8f6f885e: No bootstrap required, opened a new log
I20260924 07:10:23.766108 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:10:23.766160 26402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af842177770d4d77bf63f3abe19aec14: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:23.766594 26254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.749557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:34077 (local address 127.0.0.1:54584)
0924 07:10:23.749646 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.749659 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:23.749724 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:23.750278 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:23.750281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:23.750439 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:23.750527 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:23.750534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.752860 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.752863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:23.753501 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:23.753527 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.754229 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.754233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:23.754317 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.754880 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:23.754900 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:23.763099 (+ 8199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:23.765043 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:23.765046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:23.765048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:23.765290 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:23.765612 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:23.765615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:23.765617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:23.765706 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:23.766119 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:23.766134 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:23.766231 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 07:10:23.766308 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 07:10:23.766452 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 07:10:23.766927 26402 log.cc:824] T 00000000000000000000000000000000 P af842177770d4d77bf63f3abe19aec14: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:23.767167 26407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.749696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54584 (local address 127.3.217.60:34077)
0924 07:10:23.750059 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.750068 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.750081 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:10:23.750111 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.750113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.750153 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.751281 (+ 1128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.751368 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.752774 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.753730 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.754108 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.754503 (+ 395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.763238 (+ 8735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:23.763241 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:23.763244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:23.763267 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:23.764946 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.765420 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.765423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.765425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.765508 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.765813 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.765816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.765818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.766012 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:23.766829 (+ 817us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:23.766899 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 07:10:23.767007 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":304,"thread_start_us":131,"threads_started":1}
I20260924 07:10:23.767355 26225 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38516
I20260924 07:10:23.769830 26251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb: Bootstrap starting.
I20260924 07:10:23.769870 26326 raft_consensus.cc:348] T 00000000000000000000000000000000 P d541f35e0ebe47f2b256277d8f6f885e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42411 } } peers { permanent_uuid: "d541f35e0ebe47f2b256277d8f6f885e" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 } } peers { permanent_uuid: "af842177770d4d77bf63f3abe19aec14" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } }
I20260924 07:10:23.770241 26326 raft_consensus.cc:729] T 00000000000000000000000000000000 P d541f35e0ebe47f2b256277d8f6f885e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d541f35e0ebe47f2b256277d8f6f885e, State: Initialized, Role: FOLLOWER
I20260924 07:10:23.770581 26326 consensus_queue.cc:260] T 00000000000000000000000000000000 P d541f35e0ebe47f2b256277d8f6f885e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42411 } } peers { permanent_uuid: "d541f35e0ebe47f2b256277d8f6f885e" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 } } peers { permanent_uuid: "af842177770d4d77bf63f3abe19aec14" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } }
I20260924 07:10:23.771312 26411 sys_catalog.cc:455] T 00000000000000000000000000000000 P d541f35e0ebe47f2b256277d8f6f885e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42411 } } peers { permanent_uuid: "d541f35e0ebe47f2b256277d8f6f885e" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 } } peers { permanent_uuid: "af842177770d4d77bf63f3abe19aec14" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } } }
I20260924 07:10:23.771562 26411 sys_catalog.cc:458] T 00000000000000000000000000000000 P d541f35e0ebe47f2b256277d8f6f885e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:10:23.772090 26326 sys_catalog.cc:564] T 00000000000000000000000000000000 P d541f35e0ebe47f2b256277d8f6f885e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:23.773007 26251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:23.773802 26251 log.cc:824] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:23.774789 26402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af842177770d4d77bf63f3abe19aec14: No bootstrap required, opened a new log
I20260924 07:10:23.775593 26251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb: No bootstrap required, opened a new log
W20260924 07:10:23.777490 26423 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d541f35e0ebe47f2b256277d8f6f885e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:10:23.777622 26423 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:10:23.778327 26251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42411 } } peers { permanent_uuid: "d541f35e0ebe47f2b256277d8f6f885e" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 } } peers { permanent_uuid: "af842177770d4d77bf63f3abe19aec14" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } }
I20260924 07:10:23.778558 26251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d08fa743f614c8f8c7e3f500a89e7bb, State: Initialized, Role: FOLLOWER
I20260924 07:10:23.778853 26251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42411 } } peers { permanent_uuid: "d541f35e0ebe47f2b256277d8f6f885e" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 } } peers { permanent_uuid: "af842177770d4d77bf63f3abe19aec14" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } }
I20260924 07:10:23.779139 26402 raft_consensus.cc:348] T 00000000000000000000000000000000 P af842177770d4d77bf63f3abe19aec14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42411 } } peers { permanent_uuid: "d541f35e0ebe47f2b256277d8f6f885e" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 } } peers { permanent_uuid: "af842177770d4d77bf63f3abe19aec14" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } }
I20260924 07:10:23.779335 26402 raft_consensus.cc:729] T 00000000000000000000000000000000 P af842177770d4d77bf63f3abe19aec14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af842177770d4d77bf63f3abe19aec14, State: Initialized, Role: FOLLOWER
I20260924 07:10:23.779608 26402 consensus_queue.cc:260] T 00000000000000000000000000000000 P af842177770d4d77bf63f3abe19aec14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42411 } } peers { permanent_uuid: "d541f35e0ebe47f2b256277d8f6f885e" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 } } peers { permanent_uuid: "af842177770d4d77bf63f3abe19aec14" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } }
I20260924 07:10:23.780053 26251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:23.780107 26225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 07:10:23.780213 26225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:23.780640 26402 sys_catalog.cc:564] T 00000000000000000000000000000000 P af842177770d4d77bf63f3abe19aec14 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:23.781448 26225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42411 } } peers { permanent_uuid: "d541f35e0ebe47f2b256277d8f6f885e" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 } } peers { permanent_uuid: "af842177770d4d77bf63f3abe19aec14" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } }
I20260924 07:10:23.782366 26225 leader_election.cc:288] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb [CANDIDATE]: Term 1 election: Requested vote from peers d541f35e0ebe47f2b256277d8f6f885e (127.3.217.61:39877), af842177770d4d77bf63f3abe19aec14 (127.3.217.60:34077)
I20260924 07:10:23.783421 26373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "af842177770d4d77bf63f3abe19aec14"
I20260924 07:10:23.783854 26373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af842177770d4d77bf63f3abe19aec14 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:23.785825 26373 raft_consensus.cc:2454] T 00000000000000000000000000000000 P af842177770d4d77bf63f3abe19aec14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d08fa743f614c8f8c7e3f500a89e7bb in term 1.
I20260924 07:10:23.785877 26300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d541f35e0ebe47f2b256277d8f6f885e"
I20260924 07:10:23.786108 26300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d541f35e0ebe47f2b256277d8f6f885e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:23.786285 26202 leader_election.cc:302] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d08fa743f614c8f8c7e3f500a89e7bb, af842177770d4d77bf63f3abe19aec14; no voters:
I20260924 07:10:23.787815 26426 sys_catalog.cc:455] T 00000000000000000000000000000000 P af842177770d4d77bf63f3abe19aec14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42411 } } peers { permanent_uuid: "d541f35e0ebe47f2b256277d8f6f885e" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 } } peers { permanent_uuid: "af842177770d4d77bf63f3abe19aec14" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } } }
I20260924 07:10:23.787932 26426 sys_catalog.cc:458] T 00000000000000000000000000000000 P af842177770d4d77bf63f3abe19aec14 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:10:23.787995 26300 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d541f35e0ebe47f2b256277d8f6f885e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d08fa743f614c8f8c7e3f500a89e7bb in term 1.
I20260924 07:10:23.788136 26425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42411 } } peers { permanent_uuid: "d541f35e0ebe47f2b256277d8f6f885e" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 } } peers { permanent_uuid: "af842177770d4d77bf63f3abe19aec14" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } } }
I20260924 07:10:23.788231 26425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:10:23.788301 26425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:23.788383 26425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb [term 1 LEADER]: Becoming Leader. State: Replica: 5d08fa743f614c8f8c7e3f500a89e7bb, State: Running, Role: LEADER
I20260924 07:10:23.788787 26425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42411 } } peers { permanent_uuid: "d541f35e0ebe47f2b256277d8f6f885e" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 } } peers { permanent_uuid: "af842177770d4d77bf63f3abe19aec14" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } }
I20260924 07:10:23.776070 26258 master_runner.cc:186] Error getting master registration for 127.3.217.62:42411: OK, instance_id { permanent_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" instance_seqno: 1790233823279488 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:10:23.789569 26258 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:10:23.790661 26425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d08fa743f614c8f8c7e3f500a89e7bb. Latest consensus state: current_term: 1 leader_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42411 } } peers { permanent_uuid: "d541f35e0ebe47f2b256277d8f6f885e" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 } } peers { permanent_uuid: "af842177770d4d77bf63f3abe19aec14" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } } }
I20260924 07:10:23.790760 26425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:23.793494 26183 master_runner.cc:263] No leader master found from master 5d08fa743f614c8f8c7e3f500a89e7bb
I20260924 07:10:23.793563 26183 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 07:10:23.794656 26442 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:10:23.794727 26442 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:10:23.795286 26332 master_runner.cc:263] No leader master found from master af842177770d4d77bf63f3abe19aec14
I20260924 07:10:23.795341 26332 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 07:10:23.797353 26448 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af842177770d4d77bf63f3abe19aec14: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:10:23.797441 26448 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:10:23.801110 26457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:23.801502 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.785726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38524 (local address 127.3.217.62:42411)
0924 07:10:23.785856 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.785861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.792427 (+ 6566us) server_negotiation.cc:407] Connection header received
0924 07:10:23.792624 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.792627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.792673 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.792721 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.793790 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.794541 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.795717 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.796060 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.796130 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.797910 (+ 1780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:23.797914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:23.797917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:23.797942 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:23.799606 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.800096 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.800100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.800102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.800185 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.800508 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.800512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.800514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.800720 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:23.800817 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:23.801224 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 07:10:23.801338 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:10:23.802049 26457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:23.806457 26333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.789976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47128 (local address 127.3.217.61:39877)
0924 07:10:23.790090 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.790094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.798054 (+ 7960us) server_negotiation.cc:407] Connection header received
0924 07:10:23.798465 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.798469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.798516 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.798563 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.799317 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.800022 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.801087 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.801380 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.801456 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.803123 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:23.803126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:23.803129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:23.803152 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:23.804763 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.805228 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.805232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.805233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.805309 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.805586 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.805589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.805591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.805781 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:23.805870 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:23.806194 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 07:10:23.806307 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:10:23.811208 26373 raft_consensus.cc:1261] T 00000000000000000000000000000000 P af842177770d4d77bf63f3abe19aec14 [term 1 FOLLOWER]: Refusing update from remote peer 5d08fa743f614c8f8c7e3f500a89e7bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:10:23.811483 26407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.791774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54596 (local address 127.3.217.60:34077)
0924 07:10:23.791886 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.791890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.802107 (+ 10217us) server_negotiation.cc:407] Connection header received
0924 07:10:23.802264 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.802267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.802309 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.802356 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.803138 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.803934 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.805030 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.805655 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.805755 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.807881 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:23.807885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:23.807888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:23.807914 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:23.809564 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.810124 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.810128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.810130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.810220 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:23.810541 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:23.810544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:23.810546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:23.810759 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:23.810859 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:23.811207 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 07:10:23.811343 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:10:23.812119 26425 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "af842177770d4d77bf63f3abe19aec14" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:10:23.813109 26300 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d541f35e0ebe47f2b256277d8f6f885e [term 1 FOLLOWER]: Refusing update from remote peer 5d08fa743f614c8f8c7e3f500a89e7bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:10:23.815652 26426 sys_catalog.cc:455] T 00000000000000000000000000000000 P af842177770d4d77bf63f3abe19aec14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d08fa743f614c8f8c7e3f500a89e7bb. Latest consensus state: current_term: 1 leader_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42411 } } peers { permanent_uuid: "d541f35e0ebe47f2b256277d8f6f885e" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 } } peers { permanent_uuid: "af842177770d4d77bf63f3abe19aec14" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } } }
I20260924 07:10:23.815757 26426 sys_catalog.cc:458] T 00000000000000000000000000000000 P af842177770d4d77bf63f3abe19aec14 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:10:23.815856 26425 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d541f35e0ebe47f2b256277d8f6f885e" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:10:23.818624 26426 sys_catalog.cc:455] T 00000000000000000000000000000000 P af842177770d4d77bf63f3abe19aec14 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42411 } } peers { permanent_uuid: "d541f35e0ebe47f2b256277d8f6f885e" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 } } peers { permanent_uuid: "af842177770d4d77bf63f3abe19aec14" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } } }
I20260924 07:10:23.818720 26426 sys_catalog.cc:458] T 00000000000000000000000000000000 P af842177770d4d77bf63f3abe19aec14 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:10:23.818576 26454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42411 } } peers { permanent_uuid: "d541f35e0ebe47f2b256277d8f6f885e" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 } } peers { permanent_uuid: "af842177770d4d77bf63f3abe19aec14" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } } }
I20260924 07:10:23.818877 26454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:23.819000 26463 mvcc.cc:205] Tried to move back new op lower bound from 7332797742322835456 to 7332797742242721792. Current Snapshot: MvccSnapshot[applied={T|T < 7332797742322835456}]
I20260924 07:10:23.819044 26461 mvcc.cc:205] Tried to move back new op lower bound from 7332797742322835456 to 7332797742242721792. Current Snapshot: MvccSnapshot[applied={T|T < 7332797742322835456}]
I20260924 07:10:23.821143 26411 sys_catalog.cc:455] T 00000000000000000000000000000000 P d541f35e0ebe47f2b256277d8f6f885e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d08fa743f614c8f8c7e3f500a89e7bb. Latest consensus state: current_term: 1 leader_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42411 } } peers { permanent_uuid: "d541f35e0ebe47f2b256277d8f6f885e" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 } } peers { permanent_uuid: "af842177770d4d77bf63f3abe19aec14" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } } }
I20260924 07:10:23.821261 26411 sys_catalog.cc:458] T 00000000000000000000000000000000 P d541f35e0ebe47f2b256277d8f6f885e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:10:23.822116 26457 catalog_manager.cc:1347] Generated new cluster ID: 436e1d5747dd4633b821bcd38a494931
I20260924 07:10:23.822175 26457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:23.822649 26425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42411 } } peers { permanent_uuid: "d541f35e0ebe47f2b256277d8f6f885e" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 } } peers { permanent_uuid: "af842177770d4d77bf63f3abe19aec14" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } } }
I20260924 07:10:23.822751 26425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:23.823367 26411 sys_catalog.cc:455] T 00000000000000000000000000000000 P d541f35e0ebe47f2b256277d8f6f885e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d08fa743f614c8f8c7e3f500a89e7bb" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42411 } } peers { permanent_uuid: "d541f35e0ebe47f2b256277d8f6f885e" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 39877 } } peers { permanent_uuid: "af842177770d4d77bf63f3abe19aec14" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 34077 } } }
I20260924 07:10:23.823470 26411 sys_catalog.cc:458] T 00000000000000000000000000000000 P d541f35e0ebe47f2b256277d8f6f885e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:10:23.841357 26457 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:23.842486 26457 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:23.854919 26457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d08fa743f614c8f8c7e3f500a89e7bb: Generated new TSK 0
I20260924 07:10:23.856509 26457 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:10:23.886701 26407 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:23.884587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54608 (local address 127.3.217.60:34077)
0924 07:10:23.884693 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.884698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.885753 (+ 1055us) server_negotiation.cc:407] Connection header received
0924 07:10:23.885943 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.885946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.885990 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.886036 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.886579 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54608: BlockingRecv error: recv got EOF from 127.0.0.1:54608 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260924 07:10:23.888005 26333 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:23.885181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.3.217.61:39877)
0924 07:10:23.885280 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.885284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.887137 (+ 1853us) server_negotiation.cc:407] Connection header received
0924 07:10:23.887350 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.887353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.887397 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.887444 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.887902 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47136: BlockingRecv error: recv got EOF from 127.0.0.1:47136 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 07:10:23.889719 26253 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:23.888594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38538 (local address 127.3.217.62:42411)
0924 07:10:23.888691 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.888695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.888900 (+ 205us) server_negotiation.cc:407] Connection header received
0924 07:10:23.889083 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.889086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.889128 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.889172 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.889614 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38538: BlockingRecv error: recv got EOF from 127.0.0.1:38538 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260924 07:10:23.897218 26333 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:23.895885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47148 (local address 127.3.217.61:39877)
0924 07:10:23.895977 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.895981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.896380 (+ 399us) server_negotiation.cc:407] Connection header received
0924 07:10:23.896568 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.896571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.896633 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.896678 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.897128 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47148: BlockingRecv error: recv got EOF from 127.0.0.1:47148 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 07:10:23.901757 26333 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:23.900459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47158 (local address 127.3.217.61:39877)
0924 07:10:23.900545 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.900549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.901010 (+ 461us) server_negotiation.cc:407] Connection header received
0924 07:10:23.901162 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.901165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.901207 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.901249 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.901678 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47158: BlockingRecv error: recv got EOF from 127.0.0.1:47158 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260924 07:10:23.906913 26333 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:23.905441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47170 (local address 127.3.217.61:39877)
0924 07:10:23.905539 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.905543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.906124 (+ 581us) server_negotiation.cc:407] Connection header received
0924 07:10:23.906288 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.906291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.906332 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.906376 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.906831 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47170: BlockingRecv error: recv got EOF from 127.0.0.1:47170 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260924 07:10:23.913971 26333 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:23.912617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47186 (local address 127.3.217.61:39877)
0924 07:10:23.912738 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.912742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.913202 (+ 460us) server_negotiation.cc:407] Connection header received
0924 07:10:23.913361 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.913364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.913404 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.913447 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.913884 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47186: BlockingRecv error: recv got EOF from 127.0.0.1:47186 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 07:10:23.925138 26333 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:23.923763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47190 (local address 127.3.217.61:39877)
0924 07:10:23.923854 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.923858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.924332 (+ 474us) server_negotiation.cc:407] Connection header received
0924 07:10:23.924512 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.924515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.924559 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.924604 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.925053 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47190: BlockingRecv error: recv got EOF from 127.0.0.1:47190 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 07:10:23.946457 26333 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:23.944469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47198 (local address 127.3.217.61:39877)
0924 07:10:23.944586 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.944590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.945532 (+ 942us) server_negotiation.cc:407] Connection header received
0924 07:10:23.945722 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.945725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.945773 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.945821 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.946345 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47198: BlockingRecv error: recv got EOF from 127.0.0.1:47198 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 07:10:23.982146 26333 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:23.980860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47210 (local address 127.3.217.61:39877)
0924 07:10:23.980983 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.980987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.981213 (+ 226us) server_negotiation.cc:407] Connection header received
0924 07:10:23.981400 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.981404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.981463 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.981585 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.982027 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47210: BlockingRecv error: recv got EOF from 127.0.0.1:47210 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 07:10:24.049314 26333 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:24.048099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47218 (local address 127.3.217.61:39877)
0924 07:10:24.048259 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:24.048265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:24.048387 (+ 122us) server_negotiation.cc:407] Connection header received
0924 07:10:24.048529 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:24.048531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:24.048575 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:24.048641 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:24.049096 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47218: BlockingRecv error: recv got EOF from 127.0.0.1:47218 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 07:10:24.180408 26333 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:24.179114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47228 (local address 127.3.217.61:39877)
0924 07:10:24.179306 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:24.179312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:24.179412 (+ 100us) server_negotiation.cc:407] Connection header received
0924 07:10:24.179598 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:24.179602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:24.179659 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:24.179742 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:24.180283 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47228: BlockingRecv error: recv got EOF from 127.0.0.1:47228 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260924 07:10:24.439831 26333 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:24.438669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47232 (local address 127.3.217.61:39877)
0924 07:10:24.438829 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:24.438835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:24.438855 (+ 20us) server_negotiation.cc:407] Connection header received
0924 07:10:24.439035 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:24.439039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:24.439091 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:24.439170 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:24.439665 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47232: BlockingRecv error: recv got EOF from 127.0.0.1:47232 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 07:10:24.779584 26423 catalog_manager.cc:1259] Loaded cluster ID: 436e1d5747dd4633b821bcd38a494931
I20260924 07:10:24.779742 26423 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d541f35e0ebe47f2b256277d8f6f885e: loading cluster ID for follower catalog manager: success
I20260924 07:10:24.782172 26423 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d541f35e0ebe47f2b256277d8f6f885e: acquiring CA information for follower catalog manager: success
I20260924 07:10:24.783776 26423 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d541f35e0ebe47f2b256277d8f6f885e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:10:24.798698 26448 catalog_manager.cc:1259] Loaded cluster ID: 436e1d5747dd4633b821bcd38a494931
I20260924 07:10:24.798784 26448 catalog_manager.cc:1552] T 00000000000000000000000000000000 P af842177770d4d77bf63f3abe19aec14: loading cluster ID for follower catalog manager: success
I20260924 07:10:24.800453 26448 catalog_manager.cc:1574] T 00000000000000000000000000000000 P af842177770d4d77bf63f3abe19aec14: acquiring CA information for follower catalog manager: success
I20260924 07:10:24.801579 26448 catalog_manager.cc:1602] T 00000000000000000000000000000000 P af842177770d4d77bf63f3abe19aec14: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:10:24.957731 26533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:24.953921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47238 (local address 127.3.217.61:39877)
0924 07:10:24.954232 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:24.954238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:24.954270 (+ 32us) server_negotiation.cc:407] Connection header received
0924 07:10:24.954331 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:24.954335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:24.954457 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:24.954558 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:10:24.954824 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:24.955363 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:24.956249 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:24.956486 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:24.956581 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:24.956892 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:10:24.957309 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:10:24.957423 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:24.957497 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 07:10:24.957555 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":92,"threads_started":1}
I20260924 07:10:24.965495 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:24.961448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54618 (local address 127.3.217.60:34077)
0924 07:10:24.961755 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:24.961760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:24.961789 (+ 29us) server_negotiation.cc:407] Connection header received
0924 07:10:24.961883 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:24.961886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:24.961987 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:24.962093 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:10:24.962333 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:24.962918 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:24.963770 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:24.963995 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:24.964091 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:24.964690 (+ 599us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:10:24.965098 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:10:24.965204 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:24.965288 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 07:10:24.965343 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":81,"threads_started":1}
I20260924 07:10:24.972746 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:24.969057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54626 (local address 127.3.217.60:34077)
0924 07:10:24.969185 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:24.969189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:24.969384 (+ 195us) server_negotiation.cc:407] Connection header received
0924 07:10:24.969706 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:24.969710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:24.969755 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:24.969825 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:10:24.970094 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:24.970707 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:24.971662 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:24.971976 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:24.972036 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:24.972369 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:10:24.972429 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:10:24.972485 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:24.972582 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 07:10:24.972630 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:10:24.972908 26533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:24.968627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47252 (local address 127.3.217.61:39877)
0924 07:10:24.968927 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:24.968932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:24.968953 (+ 21us) server_negotiation.cc:407] Connection header received
0924 07:10:24.969082 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:24.969085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:24.969130 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:24.969192 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:10:24.969547 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:24.970244 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:24.971558 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:24.971894 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:24.971988 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:24.972349 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:10:24.972451 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:10:24.972515 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:24.972668 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 07:10:24.972743 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260924 07:10:24.973587 26547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:24.968741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38554 (local address 127.3.217.62:42411)
0924 07:10:24.968972 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:24.968976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:24.968992 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:10:24.969169 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:24.969173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:24.969302 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:24.969453 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:10:24.969674 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:24.970393 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:24.971634 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:24.971889 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:24.971985 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:24.972355 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:10:24.972761 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:10:24.973280 (+ 519us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:24.973314 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 07:10:24.973432 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":72,"threads_started":1}
I20260924 07:10:24.976773 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 26183
I20260924 07:10:24.983273 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 26258
I20260924 07:10:24.990316 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 26332
2026-09-24T07:10:24Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2982 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:10:25 dist-test-slave-2p7j krb5kdc[26550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:10:25 dist-test-slave-2p7j krb5kdc[26550](info): set up 2 sockets
Sep 24 07:10:25 dist-test-slave-2p7j krb5kdc[26550](info): commencing operation
krb5kdc: starting...
W20260924 07:10:25.968431 3940 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/krb5kdc/test-user.keytab.
Sep 24 07:10:26 dist-test-slave-2p7j krb5kdc[26550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
Entry for principal kudu/127.3.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab.
I20260924 07:10:26.088537 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:42791
--webserver_interface=127.3.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:42791,127.3.217.61:33483,127.3.217.60:40205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:26.191955 26563 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:10:26.192212 26563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:26.192245 26563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:26.192312 26563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:26.196168 26563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:26.196216 26563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:26.196242 26563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:26.196264 26563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:26.196291 26563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:26.201370 26563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:42791,127.3.217.61:33483,127.3.217.60:40205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.62.keytab
--principal=kudu/127.3.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.62:42791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:10:26.202277 26563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:26.203193 26563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:10:26.213160 26563 server_base.cc:1034] running on GCE node
W20260924 07:10:26.213119 26569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:26.213148 26568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:26.213119 26571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:26.213863 26563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:26.214892 26563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:26.216051 26563 hybrid_clock.cc:648] HybridClock initialized: now 1790233826216032 us; error 34 us; skew 500 ppm
Sep 24 07:10:26 dist-test-slave-2p7j krb5kdc[26550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233826, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:26.218514 26563 init.cc:377] Logged in from keytab as kudu/127.3.217.62@KRBTEST.COM (short username kudu)
I20260924 07:10:26.219287 26563 webserver.cc:460] Webserver started at http://127.3.217.62:39251/ using document root <none> and password file <none>
I20260924 07:10:26.219821 26563 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:26.219889 26563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:26.220039 26563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:26.221870 26563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-0/data/instance:
uuid: "b9e9c107c4bb48a6ad2497c1b04e033f"
format_stamp: "Formatted at 2026-09-24 07:10:26 on dist-test-slave-2p7j"
server_key: "f72a19b40031d2ccbd7ceee3e4aa2119"
server_key_iv: "0a1abfade20892d18f78a6abdd6d25f4"
server_key_version: "encryptionkey@0"
I20260924 07:10:26.222327 26563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-0/wal/instance:
uuid: "b9e9c107c4bb48a6ad2497c1b04e033f"
format_stamp: "Formatted at 2026-09-24 07:10:26 on dist-test-slave-2p7j"
server_key: "f72a19b40031d2ccbd7ceee3e4aa2119"
server_key_iv: "0a1abfade20892d18f78a6abdd6d25f4"
server_key_version: "encryptionkey@0"
I20260924 07:10:26.226150 26563 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 07:10:26.228858 26578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:26.230104 26563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:10:26.230228 26563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-0/wal
uuid: "b9e9c107c4bb48a6ad2497c1b04e033f"
format_stamp: "Formatted at 2026-09-24 07:10:26 on dist-test-slave-2p7j"
server_key: "f72a19b40031d2ccbd7ceee3e4aa2119"
server_key_iv: "0a1abfade20892d18f78a6abdd6d25f4"
server_key_version: "encryptionkey@0"
I20260924 07:10:26.230330 26563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:26.249790 26563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:26.253074 26563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:26.253250 26563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:26.260303 26563 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.62:42791
I20260924 07:10:26.260800 26630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.62:42791 every 8 connection(s)
I20260924 07:10:26.261188 26563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:26.263945 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 26563
I20260924 07:10:26.264109 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-0/wal/instance
I20260924 07:10:26.264437 3940 external_mini_cluster.cc:1428] Setting key dd00339e2a1bf8e69756c4c9ce800b33
I20260924 07:10:26.264927 26631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:26.269294 26631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42791 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.217.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.61@KRBTEST.COM" created.
Sep 24 07:10:26 dist-test-slave-2p7j krb5kdc[26550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233826, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:10:26.287609 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.271538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42791 (local address 127.0.0.1:60526)
0924 07:10:26.272031 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.272067 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:26.272868 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:26.273940 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:26.273948 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:26.274594 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:26.275106 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:26.275131 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.277130 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.277133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:26.278171 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:26.278186 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.279128 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.279132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:26.279232 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.280300 (+ 1068us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:26.280318 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:26.282263 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:26.284529 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:26.284536 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:26.284547 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:26.284844 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:26.285155 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:26.285158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:26.285160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:26.285300 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:26.285938 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:26.285943 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:26.286071 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 07:10:26.286213 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 07:10:26.286577 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":128,"spinlock_wait_cycles":190848,"thread_start_us":195,"threads_started":1}
I20260924 07:10:26.287609 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.271538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60526 (local address 127.3.217.62:42791)
0924 07:10:26.271995 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.272012 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.272153 (+ 141us) server_negotiation.cc:407] Connection header received
0924 07:10:26.273444 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.273448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.273636 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.273789 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.275442 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.276967 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.278353 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.278949 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.279120 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.282463 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:26.282470 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:26.282484 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:26.282509 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:26.284380 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.284959 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.284964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.284965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.285071 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.285380 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.285384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.285386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.285840 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:26.285954 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:26.286288 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 07:10:26.286638 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":155,"server-negotiator.queue_time_us":327,"thread_start_us":166,"threads_started":1}
I20260924 07:10:26.290015 26631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:10:26.290941 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.290657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:33483 (local address 127.0.0.1:33134)
0924 07:10:26.290818 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.290882 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.61:33483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 07:10:26.291145 26582 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.217.61:33483: connect: Connection refused (error 111)
I20260924 07:10:26.292232 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.292020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:33483 (local address 127.0.0.1:33148)
0924 07:10:26.292134 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.292181 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.61:33483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 07:10:26.292557 26631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.61:33483: Network error: Client connection negotiation failed: client connection to 127.3.217.61:33483: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.61.keytab.
Entry for principal kudu/127.3.217.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.61.keytab.
I20260924 07:10:26.294770 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.61:33483
--webserver_interface=127.3.217.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.61.keytab
--principal=kudu/127.3.217.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:42791,127.3.217.61:33483,127.3.217.60:40205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:10:26.341864 26631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 } attempt: 1
I20260924 07:10:26.342653 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.342391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:33483 (local address 127.0.0.1:53214)
0924 07:10:26.342529 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.342584 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.61:33483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 07:10:26.343376 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.343123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:33483 (local address 127.0.0.1:53222)
0924 07:10:26.343278 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.343323 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.61:33483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260924 07:10:26.343544 26631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.61:33483: Network error: Client connection negotiation failed: client connection to 127.3.217.61:33483: connect: Connection refused (error 111)
W20260924 07:10:26.399271 26638 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:10:26.399536 26638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:26.399569 26638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:26.399610 26638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:26.403358 26638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:26.403401 26638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:26.403419 26638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:26.403434 26638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:26.403446 26638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:26.408340 26638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:42791,127.3.217.61:33483,127.3.217.60:40205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.61.keytab
--principal=kudu/127.3.217.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.61:33483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.217.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:10:26.409192 26638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:26.410022 26638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:10:26.411686 26631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 } attempt: 2
I20260924 07:10:26.412356 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.412104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:33483 (local address 127.0.0.1:53236)
0924 07:10:26.412224 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.412278 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.61:33483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 07:10:26.413079 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.412849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:33483 (local address 127.0.0.1:53238)
0924 07:10:26.412943 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.413023 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.61:33483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 07:10:26.413213 26631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.61:33483: Network error: Client connection negotiation failed: client connection to 127.3.217.61:33483: connect: Connection refused (error 111)
W20260924 07:10:26.417652 26644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:26.417698 26643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:26.417758 26646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:26.418017 26638 server_base.cc:1034] running on GCE node
I20260924 07:10:26.418494 26638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:26.419998 26638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:26.421168 26638 hybrid_clock.cc:648] HybridClock initialized: now 1790233826421150 us; error 26 us; skew 500 ppm
Sep 24 07:10:26 dist-test-slave-2p7j krb5kdc[26550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233826, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:26.423638 26638 init.cc:377] Logged in from keytab as kudu/127.3.217.61@KRBTEST.COM (short username kudu)
I20260924 07:10:26.424389 26638 webserver.cc:460] Webserver started at http://127.3.217.61:39565/ using document root <none> and password file <none>
I20260924 07:10:26.424877 26638 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:26.424933 26638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:26.425097 26638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:26.426880 26638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-1/data/instance:
uuid: "06621ad2393549a1abf8e95cd14c6cd1"
format_stamp: "Formatted at 2026-09-24 07:10:26 on dist-test-slave-2p7j"
server_key: "42de2564249d7a6a100be4298c553fc3"
server_key_iv: "5e98efee2f1b90481144dfb50d8cf5fe"
server_key_version: "encryptionkey@0"
I20260924 07:10:26.427362 26638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-1/wal/instance:
uuid: "06621ad2393549a1abf8e95cd14c6cd1"
format_stamp: "Formatted at 2026-09-24 07:10:26 on dist-test-slave-2p7j"
server_key: "42de2564249d7a6a100be4298c553fc3"
server_key_iv: "5e98efee2f1b90481144dfb50d8cf5fe"
server_key_version: "encryptionkey@0"
I20260924 07:10:26.430727 26638 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:10:26.432994 26653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:26.434166 26638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:10:26.434264 26638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-1/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-1/wal
uuid: "06621ad2393549a1abf8e95cd14c6cd1"
format_stamp: "Formatted at 2026-09-24 07:10:26 on dist-test-slave-2p7j"
server_key: "42de2564249d7a6a100be4298c553fc3"
server_key_iv: "5e98efee2f1b90481144dfb50d8cf5fe"
server_key_version: "encryptionkey@0"
I20260924 07:10:26.434355 26638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:26.475718 26638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:26.478565 26638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:26.478711 26638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:26.485145 26638 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.61:33483
I20260924 07:10:26.485426 26705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.61:33483 every 8 connection(s)
I20260924 07:10:26.486027 26638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-1/data/info.pb
I20260924 07:10:26.488592 26706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:26.491314 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 26638
I20260924 07:10:26.491427 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-1/wal/instance
I20260924 07:10:26.491797 3940 external_mini_cluster.cc:1428] Setting key 68f40f4e0eb750403a21ce03a67f15e9
I20260924 07:10:26.492017 26706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42791 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.217.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.217.60@KRBTEST.COM" created.
Sep 24 07:10:26 dist-test-slave-2p7j krb5kdc[26550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233826, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.61@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:10:26.504354 26631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 } attempt: 3
I20260924 07:10:26.506947 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.493746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52746 (local address 127.3.217.62:42791)
0924 07:10:26.493920 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.493925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.494628 (+ 703us) server_negotiation.cc:407] Connection header received
0924 07:10:26.495558 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.495562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.495615 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.495711 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.496713 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.497515 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.498630 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.498986 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.499073 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.502641 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:26.502645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:26.502646 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:26.502668 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:26.504214 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.505156 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.505160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.505162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.505250 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.505550 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.505554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.505555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.505764 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:26.505900 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:26.506651 (+ 751us) server_negotiation.cc:299] Negotiation successful
0924 07:10:26.506763 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 07:10:26.508034 26709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.494070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42791 (local address 127.0.0.1:52746)
0924 07:10:26.494487 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.494533 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:26.495356 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:26.495862 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:26.495868 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:26.496172 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:26.496584 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:26.496595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.497648 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.497651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:26.498509 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:26.498522 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.499138 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.499142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:26.499294 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.499871 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:26.499886 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:26.502485 (+ 2599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:26.504471 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:26.504576 (+ 105us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:26.504594 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:26.505034 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:26.505341 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:26.505344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:26.505346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:26.505466 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:26.505907 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:26.505914 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:26.506216 (+ 302us) client_negotiation.cc:769] Sending connection context
0924 07:10:26.506539 (+ 323us) client_negotiation.cc:240] Negotiation successful
0924 07:10:26.507058 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":200,"thread_start_us":104,"threads_started":1}
I20260924 07:10:26.509740 26706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:10:26 dist-test-slave-2p7j krb5kdc[26550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233826, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for kudu/127.3.217.61@KRBTEST.COM
Sep 24 07:10:26 dist-test-slave-2p7j krb5kdc[26550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233826, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.61@KRBTEST.COM for kudu/127.3.217.61@KRBTEST.COM
I20260924 07:10:26.516645 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.504722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:33483 (local address 127.0.0.1:53252)
0924 07:10:26.504874 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.504887 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:26.504950 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:26.505615 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:26.505618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:26.505768 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:26.505872 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:26.505879 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.507423 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.507429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:26.508108 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:26.508245 (+ 137us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.509016 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.509020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:26.509158 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.509563 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:26.509578 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:26.512481 (+ 2903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:26.514579 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:26.514582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:26.514584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:26.514800 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:26.515074 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:26.515077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:26.515079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:26.515170 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:26.516257 (+ 1087us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:26.516260 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:26.516346 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 07:10:26.516438 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 07:10:26.516543 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 07:10:26.517187 26711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.504862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53252 (local address 127.3.217.61:33483)
0924 07:10:26.505157 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.505167 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.505190 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:10:26.505243 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.505248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.505450 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.505622 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.505982 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.507093 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.508397 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.508890 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.508994 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.512589 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:26.512597 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:26.512606 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:26.512625 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:26.514369 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.514887 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.514892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.514894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.514974 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.515389 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.515396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.515397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.515996 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:26.516275 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:26.516819 (+ 544us) server_negotiation.cc:299] Negotiation successful
0924 07:10:26.517007 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.217.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.60.keytab.
Entry for principal kudu/127.3.217.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.60.keytab.
I20260924 07:10:26.518885 26631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:10:26.519025 26709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.510154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:33483 (local address 127.0.0.1:53262)
0924 07:10:26.510277 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.510289 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:26.510351 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:26.510703 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:26.510705 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:26.510850 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:26.510933 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:26.510938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.511668 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.511671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:26.512330 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:26.512339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.512834 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.512838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:26.512929 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.513306 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:26.513320 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:26.514701 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:26.517716 (+ 3015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:26.517718 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:26.517720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:26.517939 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:26.518185 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:26.518188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:26.518190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:26.518279 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:26.518578 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:26.518581 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:26.518664 (+ 83us) client_negotiation.cc:769] Sending connection context
0924 07:10:26.518772 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 07:10:26.518887 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 07:10:26.519423 26712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.510301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53262 (local address 127.3.217.61:33483)
0924 07:10:26.510531 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.510536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.510544 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:10:26.510566 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.510568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.510596 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.510657 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.511044 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.511581 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.512435 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.512732 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.512821 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.514830 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:26.514832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:26.514834 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:26.514857 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:26.517613 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.518031 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.518034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.518035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.518108 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.518354 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.518356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.518357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.518506 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:26.518574 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:26.519213 (+ 639us) server_negotiation.cc:299] Negotiation successful
0924 07:10:26.519299 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":117,"threads_started":1}
I20260924 07:10:26.519728 3940 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
/tmp/dist-test-taskSXrqy4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.60:40205
--webserver_interface=127.3.217.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.60.keytab
--principal=kudu/127.3.217.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.217.20:45549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.217.62:42791,127.3.217.61:33483,127.3.217.60:40205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:10:26.520010 26706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:10:26.520541 26709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.520346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:40205 (local address 127.0.0.1:49864)
0924 07:10:26.520433 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.520493 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 07:10:26.520704 26654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.217.60:40205: connect: Connection refused (error 111)
I20260924 07:10:26.521311 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.521083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:40205 (local address 127.0.0.1:49866)
0924 07:10:26.521194 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.521247 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 07:10:26.521648 26709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.521442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:40205 (local address 127.0.0.1:49874)
0924 07:10:26.521545 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.521590 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 07:10:26.521958 26706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.60:40205: Network error: Client connection negotiation failed: client connection to 127.3.217.60:40205: connect: Connection refused (error 111)
I20260924 07:10:26.522363 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.521810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:40205 (local address 127.0.0.1:49882)
0924 07:10:26.522109 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.522305 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
W20260924 07:10:26.522478 26631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.60:40205: Network error: Client connection negotiation failed: client connection to 127.3.217.60:40205: connect: Connection refused (error 111)
I20260924 07:10:26.553647 26631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } attempt: 1
I20260924 07:10:26.554461 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.554173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:40205 (local address 127.0.0.1:49896)
0924 07:10:26.554309 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.554371 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 07:10:26.555203 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.554904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:40205 (local address 127.0.0.1:49898)
0924 07:10:26.555055 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.555149 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260924 07:10:26.555409 26631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.60:40205: Network error: Client connection negotiation failed: client connection to 127.3.217.60:40205: connect: Connection refused (error 111)
I20260924 07:10:26.571211 26706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } attempt: 1
I20260924 07:10:26.571985 26709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.571713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:40205 (local address 127.0.0.1:49900)
0924 07:10:26.571841 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.571903 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 07:10:26.572723 26709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.572515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:40205 (local address 127.0.0.1:49904)
0924 07:10:26.572612 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.572660 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 07:10:26.572891 26706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.60:40205: Network error: Client connection negotiation failed: client connection to 127.3.217.60:40205: connect: Connection refused (error 111)
W20260924 07:10:26.625103 26713 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:10:26.625345 26713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:26.625366 26713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:26.625401 26713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:26.629323 26713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:26.629375 26713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:26.629393 26713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:26.629407 26713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:26.629420 26713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:26.630577 26631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } attempt: 2
I20260924 07:10:26.631455 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.631081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:40205 (local address 127.0.0.1:49906)
0924 07:10:26.631311 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.631378 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 07:10:26.632112 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.631902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:40205 (local address 127.0.0.1:49916)
0924 07:10:26.632020 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.632060 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 07:10:26.632221 26631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.60:40205: Network error: Client connection negotiation failed: client connection to 127.3.217.60:40205: connect: Connection refused (error 111)
I20260924 07:10:26.634711 26713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.217.20:45549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.217.62:42791,127.3.217.61:33483,127.3.217.60:40205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/krb5kdc/kudu_127.3.217.60.keytab
--principal=kudu/127.3.217.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.217.60:40205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.217.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:10:26.635689 26713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:26.636592 26713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:10:26.641052 26706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } attempt: 2
I20260924 07:10:26.641654 26709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.641427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:40205 (local address 127.0.0.1:49922)
0924 07:10:26.641542 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.641592 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 07:10:26.642251 26709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.642066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:40205 (local address 127.0.0.1:49930)
0924 07:10:26.642159 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.642203 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.217.60:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 07:10:26.642381 26706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.217.60:40205: Network error: Client connection negotiation failed: client connection to 127.3.217.60:40205: connect: Connection refused (error 111)
W20260924 07:10:26.644527 26720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:26.644619 26722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:26.644685 26713 server_base.cc:1034] running on GCE node
W20260924 07:10:26.644806 26719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:26.645381 26713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:26.646934 26713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:26.648110 26713 hybrid_clock.cc:648] HybridClock initialized: now 1790233826648093 us; error 28 us; skew 500 ppm
Sep 24 07:10:26 dist-test-slave-2p7j krb5kdc[26550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233826, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:26.650676 26713 init.cc:377] Logged in from keytab as kudu/127.3.217.60@KRBTEST.COM (short username kudu)
I20260924 07:10:26.651624 26713 webserver.cc:460] Webserver started at http://127.3.217.60:43083/ using document root <none> and password file <none>
I20260924 07:10:26.652117 26713 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:26.652186 26713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:26.652361 26713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:26.654197 26713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-2/data/instance:
uuid: "0bec16968649438db3304cf883852d1b"
format_stamp: "Formatted at 2026-09-24 07:10:26 on dist-test-slave-2p7j"
server_key: "15fca29bfb99d75141241ced66b27def"
server_key_iv: "487e6d950958bb580eed84402c69618f"
server_key_version: "encryptionkey@0"
I20260924 07:10:26.654729 26713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-2/wal/instance:
uuid: "0bec16968649438db3304cf883852d1b"
format_stamp: "Formatted at 2026-09-24 07:10:26 on dist-test-slave-2p7j"
server_key: "15fca29bfb99d75141241ced66b27def"
server_key_iv: "487e6d950958bb580eed84402c69618f"
server_key_version: "encryptionkey@0"
I20260924 07:10:26.658161 26713 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260924 07:10:26.660455 26729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:26.661558 26713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 07:10:26.661665 26713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-2/data,/tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-2/wal
uuid: "0bec16968649438db3304cf883852d1b"
format_stamp: "Formatted at 2026-09-24 07:10:26 on dist-test-slave-2p7j"
server_key: "15fca29bfb99d75141241ced66b27def"
server_key_iv: "487e6d950958bb580eed84402c69618f"
server_key_version: "encryptionkey@0"
I20260924 07:10:26.661751 26713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:26.675436 26713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:26.678228 26713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:26.678390 26713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:26.685191 26713 rpc_server.cc:307] RPC server started. Bound to: 127.3.217.60:40205
I20260924 07:10:26.685482 26781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.217.60:40205 every 8 connection(s)
I20260924 07:10:26.686070 26713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-2/data/info.pb
I20260924 07:10:26.688593 26782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:26.692057 26782 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42791 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:10:26.695884 3940 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu as pid 26713
I20260924 07:10:26.696012 3940 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXrqy4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233541052846-3940-0/minicluster-data/master-2/wal/instance
I20260924 07:10:26.696277 3940 external_mini_cluster.cc:1428] Setting key 3fd688b1d1b3fd7b6b0e36c74c9857c5
Sep 24 07:10:26 dist-test-slave-2p7j krb5kdc[26550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233826, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.60@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
Sep 24 07:10:26 dist-test-slave-2p7j krb5kdc[26550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.60@KRBTEST.COM
I20260924 07:10:26.706655 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.693928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.3.217.62:42791)
0924 07:10:26.694083 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.694088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.694886 (+ 798us) server_negotiation.cc:407] Connection header received
0924 07:10:26.696323 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.696326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.696390 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.696463 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.697776 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.698308 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.700075 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.700288 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.700394 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.703186 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:26.703190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:26.703193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:26.703219 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:26.704806 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.705332 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.705335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.705336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.705408 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.705726 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.705729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.705731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.705917 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:26.705998 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:26.706374 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 07:10:26.706498 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 07:10:26.707402 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.696927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49944 (local address 127.3.217.60:40205)
0924 07:10:26.697273 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.697284 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.697306 (+ 22us) server_negotiation.cc:407] Connection header received
0924 07:10:26.697386 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.697390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.697525 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.697651 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.698317 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.699204 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.700386 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.700767 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.700870 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.703218 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:26.703225 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:26.703237 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:26.703258 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:26.705113 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.705493 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.705498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.705500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.705608 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.705852 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.705855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.705857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.706163 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:26.706351 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:26.706513 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 07:10:26.706899 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":242,"thread_start_us":156,"threads_started":1}
I20260924 07:10:26.707561 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.694053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.62:42791 (local address 127.0.0.1:52748)
0924 07:10:26.694654 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.694724 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:26.696074 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:26.696636 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:26.696643 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:26.697068 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:26.697620 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:26.697639 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.698512 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.698517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:26.699931 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:26.699947 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.700424 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.700427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:26.700564 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.701111 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:26.701131 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:26.702993 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:26.704932 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:26.704939 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:26.704946 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:26.705219 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:26.705504 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:26.705507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:26.705509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:26.705643 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:26.706009 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:26.706015 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:26.706169 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 07:10:26.706318 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 07:10:26.706632 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":85,"thread_start_us":98,"threads_started":1}
I20260924 07:10:26.708650 26782 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:10:26 dist-test-slave-2p7j krb5kdc[26550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233826, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.60@KRBTEST.COM for kudu/127.3.217.61@KRBTEST.COM
Sep 24 07:10:26 dist-test-slave-2p7j krb5kdc[26550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.62@KRBTEST.COM
I20260924 07:10:26.717125 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.709105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.61:33483 (local address 127.0.0.1:53266)
0924 07:10:26.709191 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.709204 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:26.709271 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:26.709733 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:26.709736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:26.709904 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:26.710010 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:26.710017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.710920 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.710923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:26.711648 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:26.711658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.712137 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.712139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:26.712210 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.712671 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:26.712681 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:26.714083 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:26.715709 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:26.715712 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:26.715714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:26.715962 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:26.716216 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:26.716219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:26.716221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:26.716308 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:26.716699 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:26.716702 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:26.716797 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 07:10:26.716892 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 07:10:26.716988 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 07:10:26.717185 26712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.709212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53266 (local address 127.3.217.61:33483)
0924 07:10:26.709317 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.709322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.709336 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:10:26.709480 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.709535 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.709610 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.709705 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.710146 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.710805 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.711791 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.712033 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.712134 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.714213 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:26.714218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:26.714220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:26.714262 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:26.715615 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.716060 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.716063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.716064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.716135 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.716397 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.716400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.716401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.716600 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:26.716692 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:26.716958 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 07:10:26.717073 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 07:10:26.717829 26782 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:10:26.718034 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.709536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52760 (local address 127.3.217.62:42791)
0924 07:10:26.709695 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.709698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.709711 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:10:26.709776 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.709778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.709815 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.709880 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.710473 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.711070 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.712045 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.712397 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.712495 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.714893 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:26.714895 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:26.714897 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:26.714917 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:26.716315 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.716699 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.716702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.716704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.716787 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.717056 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.717060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.717061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.717274 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:26.717436 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:26.717748 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 07:10:26.717848 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Sep 24 07:10:26 dist-test-slave-2p7j krb5kdc[26550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233826, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.60@KRBTEST.COM for kudu/127.3.217.60@KRBTEST.COM
Sep 24 07:10:26 dist-test-slave-2p7j krb5kdc[26550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.217.61@KRBTEST.COM
I20260924 07:10:26.725620 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.718438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49948 (local address 127.3.217.60:40205)
0924 07:10:26.718544 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.718548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.718559 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:10:26.718589 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.718592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.718628 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.718691 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.719033 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.719497 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.720185 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.720413 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.720491 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.722211 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:26.722214 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:26.722216 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:26.722238 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:26.723838 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.724353 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.724356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.724357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.724435 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.724697 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.724700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.724701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.724944 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:26.725046 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:26.725332 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 07:10:26.725476 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":40}
I20260924 07:10:26.725623 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.718313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:40205 (local address 127.0.0.1:49948)
0924 07:10:26.718396 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.718409 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:26.718475 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:26.718715 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:26.718718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:26.718862 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:26.718938 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:26.718944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.719606 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.719609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:26.720095 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:26.720101 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.720497 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.720501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:26.720569 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.720924 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:26.721042 (+ 118us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:26.722098 (+ 1056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:26.724015 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:26.724018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:26.724019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:26.724212 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:26.724537 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:26.724541 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:26.724542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:26.724615 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:26.725078 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:26.725081 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:26.725174 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 07:10:26.725294 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 07:10:26.725424 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 07:10:26.727377 26712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.718972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53270 (local address 127.3.217.61:33483)
0924 07:10:26.719064 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.719068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.719079 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:10:26.719155 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.719158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.719192 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.719254 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.719872 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.720313 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.720981 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.721260 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.721357 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.723491 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:26.723495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:26.723513 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:26.723537 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:26.725425 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.725882 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.725886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.725888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.725956 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.726454 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.726457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.726458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.726609 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:26.726743 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:26.727130 (+ 387us) server_negotiation.cc:299] Negotiation successful
0924 07:10:26.727241 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:10:26.729373 26782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bec16968649438db3304cf883852d1b: Bootstrap starting.
I20260924 07:10:26.731359 26631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } attempt: 3
I20260924 07:10:26.731750 26782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bec16968649438db3304cf883852d1b: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:26.732501 26782 log.cc:824] T 00000000000000000000000000000000 P 0bec16968649438db3304cf883852d1b: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:26.732995 3940 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:10:26.733520 26706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } attempt: 3
I20260924 07:10:26.735056 26782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bec16968649438db3304cf883852d1b: No bootstrap required, opened a new log
I20260924 07:10:26.734098 26605 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52760
Sep 24 07:10:26 dist-test-slave-2p7j krb5kdc[26550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233826, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.62@KRBTEST.COM for kudu/127.3.217.60@KRBTEST.COM
I20260924 07:10:26.738901 26782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bec16968649438db3304cf883852d1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42791 } } peers { permanent_uuid: "06621ad2393549a1abf8e95cd14c6cd1" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 } } peers { permanent_uuid: "0bec16968649438db3304cf883852d1b" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } }
I20260924 07:10:26.739387 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.731775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49954 (local address 127.3.217.60:40205)
0924 07:10:26.732162 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.732166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.732177 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:10:26.732205 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.732208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.732240 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.732301 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.732700 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.733414 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.734327 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.734558 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.734639 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.736488 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:26.736491 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:26.736494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:26.736513 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:26.737799 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.738343 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.738347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.738348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.738415 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.738659 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.738662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.738664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.738833 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:26.738921 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:26.739154 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 07:10:26.739251 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318}
I20260924 07:10:26.739394 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.731723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:40205 (local address 127.0.0.1:49954)
0924 07:10:26.731897 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.731910 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:26.731980 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:26.732335 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:26.732338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:26.732514 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:26.732603 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:26.732608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.733511 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.733514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:26.734223 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:26.734233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.734651 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.734654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:26.734729 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.735124 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:26.735139 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:26.736334 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:26.737972 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:26.737975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:26.737977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:26.738239 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:26.738493 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:26.738496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:26.738498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:26.738572 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:26.738933 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:26.738936 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:26.739019 (+ 83us) client_negotiation.cc:769] Sending connection context
0924 07:10:26.739099 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 07:10:26.739231 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 07:10:26.739426 26782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bec16968649438db3304cf883852d1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bec16968649438db3304cf883852d1b, State: Initialized, Role: FOLLOWER
I20260924 07:10:26.740222 26782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bec16968649438db3304cf883852d1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42791 } } peers { permanent_uuid: "06621ad2393549a1abf8e95cd14c6cd1" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 } } peers { permanent_uuid: "0bec16968649438db3304cf883852d1b" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } }
I20260924 07:10:26.740789 26789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bec16968649438db3304cf883852d1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42791 } } peers { permanent_uuid: "06621ad2393549a1abf8e95cd14c6cd1" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 } } peers { permanent_uuid: "0bec16968649438db3304cf883852d1b" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } } }
I20260924 07:10:26.741074 26789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bec16968649438db3304cf883852d1b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:10:26.741569 26782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bec16968649438db3304cf883852d1b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:26.742144 26631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f: Bootstrap starting.
Sep 24 07:10:26 dist-test-slave-2p7j krb5kdc[26550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233826, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.217.61@KRBTEST.COM for kudu/127.3.217.60@KRBTEST.COM
I20260924 07:10:26.745193 26631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:26.746006 26631 log.cc:824] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:26.747618 26709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.733864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.217.60:40205 (local address 127.0.0.1:49970)
0924 07:10:26.733981 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.733994 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:26.734078 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:26.739761 (+ 5683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:26.739765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:26.739944 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:26.740040 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:26.740047 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.740969 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.740973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:26.741697 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:26.741707 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.742267 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.742270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:26.742349 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.742829 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:26.742842 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:26.744281 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:26.746156 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:26.746159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:26.746160 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:26.746390 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:26.746654 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:26.746657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:26.746659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:26.746735 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:26.747133 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:26.747136 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:26.747228 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 07:10:26.747335 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 07:10:26.747457 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 07:10:26.748106 26631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f: No bootstrap required, opened a new log
I20260924 07:10:26.748346 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.734073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49970 (local address 127.3.217.60:40205)
0924 07:10:26.739559 (+ 5486us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.739563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.739574 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:10:26.739603 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.739606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.739644 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.740142 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.740215 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.740837 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.741836 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.742139 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.742690 (+ 551us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.744410 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:26.744414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:26.744416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:26.744437 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:26.745970 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.746491 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.746494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.746496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.746567 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.746824 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.746826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.746828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.747004 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:26.747089 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:26.748100 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0924 07:10:26.748193 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5401,"thread_start_us":75,"threads_started":1}
I20260924 07:10:26.744978 26713 master_runner.cc:186] Error getting master registration for 127.3.217.62:42791: OK, instance_id { permanent_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" instance_seqno: 1790233826258439 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:10:26.748634 26713 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 07:10:26.749368 26800 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bec16968649438db3304cf883852d1b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:10:26.749436 26800 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:10:26.750869 26706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06621ad2393549a1abf8e95cd14c6cd1: Bootstrap starting.
I20260924 07:10:26.750870 26631 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42791 } } peers { permanent_uuid: "06621ad2393549a1abf8e95cd14c6cd1" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 } } peers { permanent_uuid: "0bec16968649438db3304cf883852d1b" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } }
I20260924 07:10:26.751081 26631 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9e9c107c4bb48a6ad2497c1b04e033f, State: Initialized, Role: FOLLOWER
I20260924 07:10:26.751397 26631 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42791 } } peers { permanent_uuid: "06621ad2393549a1abf8e95cd14c6cd1" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 } } peers { permanent_uuid: "0bec16968649438db3304cf883852d1b" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } }
I20260924 07:10:26.751914 26803 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42791 } } peers { permanent_uuid: "06621ad2393549a1abf8e95cd14c6cd1" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 } } peers { permanent_uuid: "0bec16968649438db3304cf883852d1b" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } } }
I20260924 07:10:26.752151 26803 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:10:26.752585 26631 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:26.752626 26605 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 07:10:26.752722 26605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:26.753198 26706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06621ad2393549a1abf8e95cd14c6cd1: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:26.753813 26605 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42791 } } peers { permanent_uuid: "06621ad2393549a1abf8e95cd14c6cd1" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 } } peers { permanent_uuid: "0bec16968649438db3304cf883852d1b" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } }
I20260924 07:10:26.753931 26706 log.cc:824] T 00000000000000000000000000000000 P 06621ad2393549a1abf8e95cd14c6cd1: Log is configured to *not* fsync() on all Append() calls
W20260924 07:10:26.755113 26584 tablet.cc:2369] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:10:26.755489 26605 leader_election.cc:288] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f [CANDIDATE]: Term 1 election: Requested vote from peers 06621ad2393549a1abf8e95cd14c6cd1 (127.3.217.61:33483), 0bec16968649438db3304cf883852d1b (127.3.217.60:40205)
I20260924 07:10:26.756085 26751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0bec16968649438db3304cf883852d1b"
I20260924 07:10:26.756281 26751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bec16968649438db3304cf883852d1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:26.757468 26751 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0bec16968649438db3304cf883852d1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9e9c107c4bb48a6ad2497c1b04e033f in term 1.
I20260924 07:10:26.757836 26579 leader_election.cc:302] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bec16968649438db3304cf883852d1b, b9e9c107c4bb48a6ad2497c1b04e033f; no voters:
I20260924 07:10:26.758086 26803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:26.758256 26803 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f [term 1 LEADER]: Becoming Leader. State: Replica: b9e9c107c4bb48a6ad2497c1b04e033f, State: Running, Role: LEADER
I20260924 07:10:26.758543 26803 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42791 } } peers { permanent_uuid: "06621ad2393549a1abf8e95cd14c6cd1" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 } } peers { permanent_uuid: "0bec16968649438db3304cf883852d1b" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } }
I20260924 07:10:26.758882 26706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06621ad2393549a1abf8e95cd14c6cd1: No bootstrap required, opened a new log
W20260924 07:10:26.759033 26815 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:10:26.759089 26815 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:10:26.760123 26803 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9e9c107c4bb48a6ad2497c1b04e033f. Latest consensus state: current_term: 1 leader_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42791 } } peers { permanent_uuid: "06621ad2393549a1abf8e95cd14c6cd1" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 } } peers { permanent_uuid: "0bec16968649438db3304cf883852d1b" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } } }
I20260924 07:10:26.760236 26803 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:26.758534 26563 master_runner.cc:186] Error getting master registration for 127.3.217.61:33483: OK, instance_id { permanent_uuid: "06621ad2393549a1abf8e95cd14c6cd1" instance_seqno: 1790233826483350 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:10:26.761586 26563 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:10:26.762567 26706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06621ad2393549a1abf8e95cd14c6cd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42791 } } peers { permanent_uuid: "06621ad2393549a1abf8e95cd14c6cd1" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 } } peers { permanent_uuid: "0bec16968649438db3304cf883852d1b" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } }
I20260924 07:10:26.762781 26706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06621ad2393549a1abf8e95cd14c6cd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06621ad2393549a1abf8e95cd14c6cd1, State: Initialized, Role: FOLLOWER
I20260924 07:10:26.763082 26706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06621ad2393549a1abf8e95cd14c6cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42791 } } peers { permanent_uuid: "06621ad2393549a1abf8e95cd14c6cd1" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 } } peers { permanent_uuid: "0bec16968649438db3304cf883852d1b" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } }
I20260924 07:10:26.763940 26824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:26.764338 26706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06621ad2393549a1abf8e95cd14c6cd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:26.764381 26680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "06621ad2393549a1abf8e95cd14c6cd1"
I20260924 07:10:26.764576 26680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06621ad2393549a1abf8e95cd14c6cd1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:26.765822 26680 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 06621ad2393549a1abf8e95cd14c6cd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9e9c107c4bb48a6ad2497c1b04e033f in term 1.
I20260924 07:10:26.767328 26824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260924 07:10:26.773520 26837 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06621ad2393549a1abf8e95cd14c6cd1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:10:26.773597 26837 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:10:26.773886 26826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06621ad2393549a1abf8e95cd14c6cd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42791 } } peers { permanent_uuid: "06621ad2393549a1abf8e95cd14c6cd1" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 } } peers { permanent_uuid: "0bec16968649438db3304cf883852d1b" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } } }
I20260924 07:10:26.773989 26826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06621ad2393549a1abf8e95cd14c6cd1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:10:26.774129 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.758077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49972 (local address 127.3.217.60:40205)
0924 07:10:26.758166 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.758170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.763145 (+ 4975us) server_negotiation.cc:407] Connection header received
0924 07:10:26.763488 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.763492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.763558 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.763606 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.766923 (+ 3317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.767590 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.768700 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.769011 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.769079 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.770729 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:26.770743 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:26.770746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:26.770771 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:26.772311 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.772761 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.772764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.772766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.772851 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.773136 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.773139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.773141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.773337 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:26.773429 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:26.773888 (+ 459us) server_negotiation.cc:299] Negotiation successful
0924 07:10:26.773990 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 07:10:26.782155 26712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.762142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53282 (local address 127.3.217.61:33483)
0924 07:10:26.762244 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.762248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.774261 (+ 12013us) server_negotiation.cc:407] Connection header received
0924 07:10:26.774421 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.774424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.774464 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.774508 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.775331 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.775988 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.776991 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.777316 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.777378 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.778884 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:26.778887 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:26.778890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:26.778912 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:26.780434 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.780891 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.780895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.780897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.780976 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.781271 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.781275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.781276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.781469 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:26.781573 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:26.781905 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 07:10:26.782007 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:10:26.786722 26751 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0bec16968649438db3304cf883852d1b [term 1 FOLLOWER]: Refusing update from remote peer b9e9c107c4bb48a6ad2497c1b04e033f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:10:26.787341 26803 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bec16968649438db3304cf883852d1b" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:10:26.787592 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.774521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52770 (local address 127.3.217.62:42791)
0924 07:10:26.774620 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.774624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.775677 (+ 1053us) server_negotiation.cc:407] Connection header received
0924 07:10:26.775826 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.775829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.775869 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.775910 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.776665 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.777358 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.782489 (+ 5131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.782746 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.782807 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.784222 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:26.784226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:26.784228 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:26.784260 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:26.785847 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.786312 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.786316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.786318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.786401 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:26.786678 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:26.786681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:26.786683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:26.786886 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:26.786993 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:26.787314 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 07:10:26.787431 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:10:26.791258 26789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bec16968649438db3304cf883852d1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9e9c107c4bb48a6ad2497c1b04e033f. Latest consensus state: current_term: 1 leader_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42791 } } peers { permanent_uuid: "06621ad2393549a1abf8e95cd14c6cd1" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 } } peers { permanent_uuid: "0bec16968649438db3304cf883852d1b" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } } }
I20260924 07:10:26.791380 26789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bec16968649438db3304cf883852d1b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:10:26.792582 26680 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 06621ad2393549a1abf8e95cd14c6cd1 [term 1 FOLLOWER]: Refusing update from remote peer b9e9c107c4bb48a6ad2497c1b04e033f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:10:26.793095 26822 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f [LEADER]: Connected to new peer: Peer: permanent_uuid: "06621ad2393549a1abf8e95cd14c6cd1" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:10:26.795528 26822 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42791 } } peers { permanent_uuid: "06621ad2393549a1abf8e95cd14c6cd1" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 } } peers { permanent_uuid: "0bec16968649438db3304cf883852d1b" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } } }
I20260924 07:10:26.795651 26822 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:26.796154 26826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06621ad2393549a1abf8e95cd14c6cd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9e9c107c4bb48a6ad2497c1b04e033f. Latest consensus state: current_term: 1 leader_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42791 } } peers { permanent_uuid: "06621ad2393549a1abf8e95cd14c6cd1" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 } } peers { permanent_uuid: "0bec16968649438db3304cf883852d1b" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } } }
I20260924 07:10:26.796260 26826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06621ad2393549a1abf8e95cd14c6cd1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:10:26.796692 26789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bec16968649438db3304cf883852d1b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42791 } } peers { permanent_uuid: "06621ad2393549a1abf8e95cd14c6cd1" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 } } peers { permanent_uuid: "0bec16968649438db3304cf883852d1b" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } } }
I20260924 07:10:26.796797 26789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bec16968649438db3304cf883852d1b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:10:26.797143 26839 mvcc.cc:205] Tried to move back new op lower bound from 7332797754511413248 to 7332797754406043648. Current Snapshot: MvccSnapshot[applied={T|T < 7332797754511413248}]
I20260924 07:10:26.797477 26841 mvcc.cc:205] Tried to move back new op lower bound from 7332797754511413248 to 7332797754406043648. Current Snapshot: MvccSnapshot[applied={T|T < 7332797754511413248}]
I20260924 07:10:26.803227 26824 catalog_manager.cc:1347] Generated new cluster ID: 207b7246bacf4349a825c4195babe134
I20260924 07:10:26.803287 26824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:26.805047 26803 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42791 } } peers { permanent_uuid: "06621ad2393549a1abf8e95cd14c6cd1" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 } } peers { permanent_uuid: "0bec16968649438db3304cf883852d1b" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } } }
I20260924 07:10:26.805150 26803 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:26.805997 26826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06621ad2393549a1abf8e95cd14c6cd1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e9c107c4bb48a6ad2497c1b04e033f" member_type: VOTER last_known_addr { host: "127.3.217.62" port: 42791 } } peers { permanent_uuid: "06621ad2393549a1abf8e95cd14c6cd1" member_type: VOTER last_known_addr { host: "127.3.217.61" port: 33483 } } peers { permanent_uuid: "0bec16968649438db3304cf883852d1b" member_type: VOTER last_known_addr { host: "127.3.217.60" port: 40205 } } }
I20260924 07:10:26.806110 26826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06621ad2393549a1abf8e95cd14c6cd1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:10:26.806967 26843 mvcc.cc:205] Tried to move back new op lower bound from 7332797754511413248 to 7332797754406043648. Current Snapshot: MvccSnapshot[applied={T|T < 7332797754511413248}]
I20260924 07:10:26.827931 26824 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:26.829068 26824 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:26.856030 26824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9e9c107c4bb48a6ad2497c1b04e033f: Generated new TSK 0
I20260924 07:10:26.856803 26824 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:10:26.880872 26712 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:26.878603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53290 (local address 127.3.217.61:33483)
0924 07:10:26.878715 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.878719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.879899 (+ 1180us) server_negotiation.cc:407] Connection header received
0924 07:10:26.880094 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.880098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.880143 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.880191 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.880755 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53290: BlockingRecv error: recv got EOF from 127.0.0.1:53290 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 07:10:26.882339 26634 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:26.879322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52772 (local address 127.3.217.62:42791)
0924 07:10:26.879425 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.879429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.881549 (+ 2120us) server_negotiation.cc:407] Connection header received
0924 07:10:26.881700 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.881704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.881749 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.881796 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.882237 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52772: BlockingRecv error: recv got EOF from 127.0.0.1:52772 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 07:10:26.884274 26786 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:26.883023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.3.217.60:40205)
0924 07:10:26.883128 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.883132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.883401 (+ 269us) server_negotiation.cc:407] Connection header received
0924 07:10:26.883584 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.883588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.883631 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.883683 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.884169 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49982: BlockingRecv error: recv got EOF from 127.0.0.1:49982 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 07:10:26.888316 26712 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:26.886873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53294 (local address 127.3.217.61:33483)
0924 07:10:26.886977 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.886981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.887416 (+ 435us) server_negotiation.cc:407] Connection header received
0924 07:10:26.887637 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.887640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.887707 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.887756 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.888224 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53294: BlockingRecv error: recv got EOF from 127.0.0.1:53294 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 07:10:26.893007 26712 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:26.891603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53300 (local address 127.3.217.61:33483)
0924 07:10:26.891702 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.891706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.892195 (+ 489us) server_negotiation.cc:407] Connection header received
0924 07:10:26.892376 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.892379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.892424 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.892469 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.892917 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53300: BlockingRecv error: recv got EOF from 127.0.0.1:53300 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 07:10:26.898233 26712 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:26.896839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53312 (local address 127.3.217.61:33483)
0924 07:10:26.896931 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.896935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.897452 (+ 517us) server_negotiation.cc:407] Connection header received
0924 07:10:26.897623 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.897625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.897671 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.897716 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.898149 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53312: BlockingRecv error: recv got EOF from 127.0.0.1:53312 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 07:10:26.906741 26712 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:26.904633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53322 (local address 127.3.217.61:33483)
0924 07:10:26.904990 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.904996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.905816 (+ 820us) server_negotiation.cc:407] Connection header received
0924 07:10:26.906021 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.906025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.906077 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.906127 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.906649 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53322: BlockingRecv error: recv got EOF from 127.0.0.1:53322 (error 108)
Metrics: {"server-negotiator.queue_time_us":248}
W20260924 07:10:26.918412 26712 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:26.916810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.3.217.61:33483)
0924 07:10:26.916912 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.916916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.917486 (+ 570us) server_negotiation.cc:407] Connection header received
0924 07:10:26.917684 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.917688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.917740 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.917793 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.918316 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53324: BlockingRecv error: recv got EOF from 127.0.0.1:53324 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 07:10:26.938433 26712 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:26.936696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53330 (local address 127.3.217.61:33483)
0924 07:10:26.936803 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.936808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.937404 (+ 596us) server_negotiation.cc:407] Connection header received
0924 07:10:26.937621 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.937624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.937688 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.937735 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.938324 (+ 589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53330: BlockingRecv error: recv got EOF from 127.0.0.1:53330 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 07:10:26.974195 26712 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:26.972876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53346 (local address 127.3.217.61:33483)
0924 07:10:26.973007 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.973013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.973314 (+ 301us) server_negotiation.cc:407] Connection header received
0924 07:10:26.973466 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.973470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.973527 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.973609 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.974078 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53346: BlockingRecv error: recv got EOF from 127.0.0.1:53346 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 07:10:27.041333 26712 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:27.040274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53354 (local address 127.3.217.61:33483)
0924 07:10:27.040419 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:27.040423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:27.040556 (+ 133us) server_negotiation.cc:407] Connection header received
0924 07:10:27.040652 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:27.040654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:27.040699 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:27.040767 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:27.041188 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53354: BlockingRecv error: recv got EOF from 127.0.0.1:53354 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 07:10:27.172411 26712 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:27.171278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53366 (local address 127.3.217.61:33483)
0924 07:10:27.171415 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:27.171419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:27.171592 (+ 173us) server_negotiation.cc:407] Connection header received
0924 07:10:27.171725 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:27.171728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:27.171774 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:27.171840 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:27.172291 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53366: BlockingRecv error: recv got EOF from 127.0.0.1:53366 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 07:10:27.431641 26712 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:27.430609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53376 (local address 127.3.217.61:33483)
0924 07:10:27.430737 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:27.430741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:27.430803 (+ 62us) server_negotiation.cc:407] Connection header received
0924 07:10:27.430917 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:27.430919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:27.430965 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:27.431032 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:27.431527 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53376: BlockingRecv error: recv got EOF from 127.0.0.1:53376 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:10:27.751168 26800 catalog_manager.cc:1259] Loaded cluster ID: 207b7246bacf4349a825c4195babe134
I20260924 07:10:27.751317 26800 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0bec16968649438db3304cf883852d1b: loading cluster ID for follower catalog manager: success
I20260924 07:10:27.753487 26800 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0bec16968649438db3304cf883852d1b: acquiring CA information for follower catalog manager: success
I20260924 07:10:27.754992 26800 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0bec16968649438db3304cf883852d1b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:10:27.775177 26837 catalog_manager.cc:1259] Loaded cluster ID: 207b7246bacf4349a825c4195babe134
I20260924 07:10:27.775318 26837 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 06621ad2393549a1abf8e95cd14c6cd1: loading cluster ID for follower catalog manager: success
I20260924 07:10:27.777422 26837 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 06621ad2393549a1abf8e95cd14c6cd1: acquiring CA information for follower catalog manager: success
I20260924 07:10:27.778663 26837 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 06621ad2393549a1abf8e95cd14c6cd1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:10:27.950117 26912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:27.945701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53378 (local address 127.3.217.61:33483)
0924 07:10:27.946061 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:27.946065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:27.946085 (+ 20us) server_negotiation.cc:407] Connection header received
0924 07:10:27.946140 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:27.946142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:27.946263 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:27.946374 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:10:27.946820 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:27.947424 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:27.948494 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:27.948753 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:27.948843 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:27.949169 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:10:27.949594 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:10:27.949685 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:27.949804 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 07:10:27.949900 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":103,"threads_started":1}
I20260924 07:10:27.957509 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:27.953806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49988 (local address 127.3.217.60:40205)
0924 07:10:27.954110 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:27.954114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:27.954130 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:10:27.954280 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:27.954283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:27.954409 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:27.954579 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:10:27.954805 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:27.955319 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:27.956180 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:27.956406 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:27.956487 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:27.956757 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:10:27.957137 (+ 380us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:10:27.957237 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:27.957305 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 07:10:27.957366 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":84,"threads_started":1}
I20260924 07:10:27.964848 26912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:27.960600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53392 (local address 127.3.217.61:33483)
0924 07:10:27.960725 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:27.960728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:27.961175 (+ 447us) server_negotiation.cc:407] Connection header received
0924 07:10:27.961319 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:27.961322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:27.961368 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:27.961436 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:10:27.961695 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:27.962279 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:27.963632 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:27.963862 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:27.963919 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:27.964344 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:10:27.964420 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:10:27.964515 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:27.964649 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 07:10:27.964713 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:10:27.964954 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:27.960431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52786 (local address 127.3.217.62:42791)
0924 07:10:27.960645 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:27.960649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:27.961045 (+ 396us) server_negotiation.cc:407] Connection header received
0924 07:10:27.961134 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:27.961137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:27.961271 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:27.961384 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:10:27.961744 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:27.962520 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:27.963397 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:27.963699 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:27.963786 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:27.964141 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:10:27.964420 (+ 279us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:10:27.964526 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:27.964776 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 07:10:27.964828 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":87,"threads_started":1}
I20260924 07:10:27.964949 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:27.960961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49998 (local address 127.3.217.60:40205)
0924 07:10:27.961089 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:27.961093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:27.961108 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:10:27.961171 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:27.961174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:27.961253 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:27.961322 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:10:27.961629 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:27.962409 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:27.963601 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:27.963901 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:27.963997 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:27.964368 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:10:27.964473 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:10:27.964526 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:27.964712 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 07:10:27.964792 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:10:27.968736 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 26563
I20260924 07:10:27.977020 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 26638
I20260924 07:10:27.983032 3940 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXrqy4/build/debug/bin/kudu with pid 26713
2026-09-24T07:10:27Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2992 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5975 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (286923 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST